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PREFACE 


RoadRunner  '98  was  a  training  research  exercise  sponsored  by  the  Air  Force  Modeling 
and  Simulation  Office  (USAF/XOC).  This  report  documents  work  performed  by  the  Air  Force 
Research  Laboratory,  Human  Effectiveness  Directorate,  Warfighter  Training  Research  Division 
(AFRL/HEA)  in  Mesa  AZ.  This  effort  was  conducted  under  Work  Unit  2743-B2-05,  Distributed 
Mission  Training.  The  Laboratory  Work  Unit  Monitor  was  Maj  Justine  Good;  the  Laboratory 
Project  Scientist  was  Dr  Peter  M.  Crane. 


The  authors  gratefully  acknowledge  assistance  in  preparation  of  this  technical  report  from 
Linda  Elliott,  Herbert  Bell,  Robbie  Robbins,  Don  Smoot,  Paul  Nielson,  J.  D.  Dennison,  Maj 
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thanks  to  the  many  individuals  and  organizations  who  cooperated  in  the  development  and 
conduct  of  the  RoadRunner  '98  exercise  with  special  thanks  to  the  participating  warfighters. 
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EXECUTIVE  SUMMARY 


RoadRunner  '98  was  a  research  exercise  designed  to  assess  the  technical  feasibility  and 
training  potential  of  Distributed  Mission  Training  (DMT).  RoadRunner  '98  was  sponsored  by 
the  Air  Force  Modeling  and  Simulation  Office  (USAF/XOC)  and  conducted  by  the  Air  Force 
Research  Laboratory,  Warfighter  Training  Research  Division  (AFRL/HEA)  and  the  Theater  Air 
Command  and  Control  Simulation  Facility  (TACCSF).  DMT  is  an  emerging  training  concept  in 
which  warfighters  will  use  advanced,  real-time  simulators  located  around  the  world  to  conduct 
mission  training  in  a  virtual  battlespace.  As  envisioned  by  Air  Combat  Command  (ACC),  DMT 
will  complement  current  squadron  training  and  large-force  exercises  by  providing  additional 
experience  in  performing  tasks  and  missions  that  are  infrequently  practiced  or  highly 
constrained.  Using  DMT,  warfighters  will  be  able  to  interact  with  each  other  and  with  computer¬ 
generated  forces  to  conduct  composite  force  missions  unconstrained  by  limitations  of  cost, 
safety,  and  security.  The  objectives  of  RoadRunner  '98  were  to  determine  whether  state-of-the- 
art  DMT  technologies  were  capable  of  supporting  composite  force  training  exercises  over  a  wide 
area  network  and  to  validate  a  research  approach  for  conducting  DMT  training  effectiveness 
evaluations.  This  report  describes  how  the  RoadRunner  '98  exercise  was  conducted,  the  data  that 
was  collected,  and  the  successes  and  shortfalls  in  meeting  exercise  objectives.* 

In  RoadRunner  '98,  F-15  and  F-16  fighter  pilots  and  Airborne  Warning  and  Control 
System  (AW ACS)  air  weapons  control  teams  flew  seven  training  missions  from  locations  in 
Arizona,  New  Mexico,  and  Oklahoma  during  the  period  of  13-17  Jul  98.  Data  collected  were 
measures  of  mission  accomplishment,  instructor  ratings  of  team  performance,  and  participant 
critiques  and  recommendations.  All  RoadRunner  '98  missions  were  completed  as  planned 
without  major  difficulties  or  delays.  Results  show  that  team  performance  on  composite  force 
missions  improved  during  RoadRunner  '98,  pilots  and  AWACS  crews  were  able  to  successfully 
complete  their  missions,  and  all  participants  rated  DMT  as  having  significant  training  potential. 
Participants  identified  several  areas  for  improvement,  notably,  visual  display  systems,  computer¬ 
generated  forces,  and  interactions  among  simulated  entities.  Data  from  RoadRunner  '98  also 
show  that  the  perceived  training  value  of  participating  in  DMT  exercises  depends  on  both  the 
capability  of  DMT  systems  to  support  a  particular  mission  or  skill  and  the  availability  of  aircraft 
training  for  that  mission  or  skill.  Recommendations  are  offered  for  further  development  of  DMT 
systems  and  for  future  DMT  exercises. 


*  Additional  information  on  the  RoadRunner  '98  exercise  is  available  in  a  classified  report: 
Cicero,  G.  D.  (1998).  Simulated  Threat  Performance  Observations  for  RoadRunner  '9<§(U). 
(AFRL-HE-AZ-TR- 1 998-0 1 00)  Mesa,  AZ:  Air  Force  Research  Laboratory. 
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ROADRUNNER  ’98:  TRAINING  EFFECTIVENESS 
IN  A  DISTRIBUTED  MISSION  TRAINING  EXERCISE 


INTRODUCTION 

RoadRunner  '98  was  an  Air  Force  exercise  designed  to  assess  the  state  of  the  art  in 
Distributed  Mission  Training  (DMT)  for  aircrew  training.  DMT  is  an  emerging  training  concept 
in  which  warfighters  enter  into  a  synthetic  battlespace  using  live,  constructive,  and  virtual 
simulations.  In  the  synthetic  battlespace,  warfighters  are  able  to  execute  complex,  composite 
force  missions  unconstrained  by  issues  of  cost,  safety,  and  security  that  restrict  current  live 
exercises.  Unlike  previous  multiplayer  simulation  studies  that  were  conducted  to  demonstrate  a 
proof-of-concept  or  to  evaluate  technology,  the  intent  of  RoadRunner  '98  was  to  assess  the 
effectiveness  of  DMT  by  bringing  operational  pilots  and  battle  managers  together  for  a  series  of 
composite  force  missions.  The  products  of  RoadRunner  '98  are  an  assessment  of  the  training 
potential  of  DMT,  recommendations  for  best  use  of  current  DMT  systems,  a  description  of 
lessons  learned,  and  recommendations  for  improving  DMT  systems.  This  report  describes  and 
summarizes  the  measures  of  training  effectiveness  collected  during  the  RoadRunner  '98  DMT 
exercise  and  recommendations  for  improving  training  effectiveness. 

RoadRunner  '98  Overview 

Development  and  conduct  of  RoadRunner  '98  was  a  combined  effort  of  many 
organizations.  RoadRunner  '98  was  sponsored  by  the  Air  Force  Modeling  and  Simulation  Office 
(USAF/XOC)  with  the  Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate’s 
Warfighter  Training  Research  Division  (AFRL/HEA)  serving  as  program  managers  and  the 
Theater  Air  Command  and  Control  Simulation  Facility  (TACCSF)  as  systems  integrators. 
Extensive  support  was  received  from  the  Training  Office  at  Air  Combat  Command’s  Deputy  for 
Operations  (ACC/DOT),  the  Air  Force  Information  Warfare  Center’s  Advanced  Combat 
Simulations  Division  (AFIWC/SAM),  the  Airborne  Warning  and  Control  Systems  (AWACS) 
training  office  at  the  552nd  Air  Control  Wing  (ACW),  the  133rd  Airborne  Air  Control  Squadron 
(AACS)  of  the  Iowa  Air  National  Guard  (IANG),  and  the  Naval  Air  Warfare  Center’s  Training 
Systems  Division  (NAWC-TSD).  The  warfighters  who  participated  in  RoadRunner  '98  were 
operational  pilots  from  the  27th  Fighter  Wing  (FW),  Cannon  AFB  NM;  the  185  and  132  FWs  of 
the  IANG;  the  33rd  Fighter  Wing,  Eglin  AFB  FL;  United  States  Air  Forces  in  Europe  (USAFE); 
and  weapons  directors  and  air  surveillance  technicians  from  the  552  ACW,  Tinker  AFB  OK. 

The  goal  of  DMT  is  to  provide  home  station  training  for  all  warfighters.  During 
RoadRunner  '98,  only  AWACS  had  on-base  simulation  systems  capable  of  participating  in  a 
DMT  exercise.  F-16  pilots  used  AFRL/HEA’s  system  of  four  F-16C  Multitask  Trainers  (MTTs) 
located  in  Mesa  AZ,  while  F-15  pilots  used  TACCSF’s  system  of  four  Boeing  Weapons  and 
Tactics  Trainers  (WTTs)  at  Kirtland  AFB  NM. 

During  the  period  of  13-17  Jul  98,  teams  of  warfighters  flew  one  familiarization  mission 
plus  seven  composite  force  missions  over  a  synthetic  Red  Flag  Training  Range.  Each  mission 
was  executed  over  a  secure,  wide-area  network  using  Distributed  Interactive  Simulation  (DIS) 
communications  protocols.  In  these  missions,  pilots  and  AWACS  teams  interacted  with  virtual 


and  constructive,  i.e.,  computer-generated,  forces  including  friendly  and  enemy  fighters, 
helicopters,  and  ground  vehicles,  plus  enemy  surface-to-air  threats.  Five  of  the  seven  composite 
force  missions  were  offensive  air-to-surface,  one  mission  was  defensive  air-to-air,  and  one  was 
Close  Air  Support  (CAS).  The  intent  of  RoadRunner  '98  was  to  provide  operational  warfighters 
with  the  opportunity  to  experience  DMT  using  state-of-the-art  systems.  Based  on  their 
experience,  warfighters  were  asked  to  identify  the  technical  successes  and  shortfalls  in  these 
systems  and  to  assess  the  potential  for  DMT  to  improve  future  Air  Force  training. 

Multiplayer  Simulation  Background 

Although  flight  simulation  has  been  a  part  of  aircrew  training  since  the  Link  “Blue  Box” 
instrument  simulators  of  World  War  II,  such  training  was  traditionally  limited  to  training  basic 
procedural  and  psychomotor  skills.  This  limited  training  reflected  the  capabilities  of  simulator 
technology  that  could  not  support  high-fidelity,  individual  training  for  tasks  other  than  basic 
skills.  As  simulation  technology  improved,  however,  the  idea  was  proposed  that  multiple 
simulators  could  be  linked  to  allow  pilots  and  other  warfighters  to  conduct  multiplayer 
engagements  in  a  virtual  battlespace  (Hapgood,  1997).  By  the  mid-1980s  technology  advanced 
to  the  point  that  it  was  possible  to  interconnect  simulators  and  conduct  team  training.  The  first 
example  of  such  training  was  the  SIMNET  project  in  which  a  number  of  tank  simulators  were 
interconnected  to  provide  collective  training  (Alluisi,  1991).  Based  on  the  success  of  the 
SIMNET  program,  the  Air  Force  Research  Laboratory  embarked  on  a  science  and  technology 
program  to  develop  simulator-based  training  technologies  and  methods  that  would  afford 
aircrews  the  same  training  opportunities  that  SIMNET  afforded  the  ground  forces. 

One  of  the  first  research  studies  which  examined  aircrew  training  in  a  team  context  took 
place  at  the  McDonnell-Douglas’s  F-15  simulator  complex  (Houck,  Thomas,  &  Bell,  1991). 
Mission-ready  F-15  pilots  and  air  weapons  controllers  received  advanced  air  combat  training  as 
three-member  teams.  The  two  pilots  flew  high-fidelity  F-15  cockpits  located  in  40-ft  domes 
while  a  Weapons  Director  operated  a  high-fidelity  display  and  communication  system.  A  full 
spectrum  of  simulated  air  combat  missions  were  flown  against  a  variety  of  enemy  forces  which 
were  either  constructive  models  or  virtual  aircraft  flown  by  pilots  from  small  player  stations. 
Teams  performed  their  normal  mission  planning  and  premission  briefing.  They  also  debriefed  as 
usual  using  an  instrumented  debriefing  system  that  provided  them  with  a  plan  view  display  of  the 
gaming  area,  full  cockpit  instrumentation,  voice  communication,  and  detailed  information  on 
weapons  effectiveness.  Participant  feedback  and  measures  of  combat  effectiveness  showed  that 
mission  performance  improved  with  training  and  that  the  warfighters  were  generally  positive 
about  the  experience.  They  rated  simulator-based  combat  training  as  superior  to  their  local  flying 
training  for  some  combat  tasks. 

Based  on  the  success  of  the  air  combat  training  program  at  McDonnell-Douglas,  a 
combat  mission  training  simulation  testbed  was  established  at  the  Air  Force  Research 
Laboratory’s  facility  in  Mesa,  AZ.  This  facility  has  been  used  to  conduct  both  engineering 
development  and  training  effectiveness  research.  One  example  is  the  Multiservice  Distributed 
Training  Testbed  (MDT2)  training  research  project  that  was  conducted  using  assets  from  all  four 
services  at  locations  across  the  country.  As  part  of  the  MDT2  project,  a  wide-area  training 
network  was  established  that  linked  Air  Force  flight  simulators  in  Mesa  AZ,  and  Navy  flight 
simulators  at  Patuxant  River  MD,  with  Army  tank  simulators  at  Ft.  Knox  KY,  and  a  Marine  fire 
support  simulator  in  San  Diego  CA  (Bell,  et  al.,  1996).  All  these  heterogeneous  virtual 
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simulators  interacted  within  a  common  synthetic  environment  that  represented  the  National 
Training  Center  at  Ft.  Irwin  CA.  These  virtual  simulators  were  integrated  with  a  manned  Tactical 
Operations  Center,  staffed  by  appropriate  Army  and  Air  Force  personnel.  In  addition,  Modular 
Semi-Automated  Forces  provided  constructive  friendly  and  enemy  forces  needed  for  a  realistic 
battalion-level  engagement.  During  MDT2,  two  Air  Force  F-16  simulators  provided  close  air 
support  to  a  battalion  task  force  with  support  from  a  Marine  forward  air  controller  (FAC)  located 
on  the  ground  and  a  Navy  airborne  FAC  flying  an  OV-IO  simulator.  Each  simulator 
communicated  using  DIS  protocols  and  shared  a  common  battlespace.  This  common  synthetic 
environment  incorporated  11  virtual  simulators  and  over  100  computer-generated  entities  to 
provide  tactical  combat  training  to  33  individual  warfighters.  Communication  and  coordination 
skills  all  improved  by  significant  margins  as  the  warfighters  from  different  services  gained 
experience  working  together.  As  these  communication  and  coordination  skills  improved,  the 
ability  to  effectively  execute  close  air  support  procedures  increased  and  the  number  of  successful 
weapons  deliveries  and  the  number  of  enemy  tanks  destroyed  increased. 

Attitude  questionnaires  and  interviews  administered  to  MDT2  trainees  from  each  service 
showed  considerable  enthusiasm  for  this  type  of  training.  Many  of  the  warfighters  felt  that  it  was 
the  best  training  they  had  ever  received  in  performing  such  difficult  joint  force  training.  Their 
attitude  is  perhaps  best  summed  up  by  the  comments  of  the  battalion  commander  who  said  it  was 
the  most  and  best  close  air  support  training  he  had  received  in  his  nearly  20-year  career. 

DMT  as  Central  Component  of  Future  Air  Force  Training 

Multiplayer  simulation  systems  have  been  used  by  the  U.  S.  Army  for  annored  vehicle 
training  and  by  the  U.  S.  Navy  for  shipboard  combat  information  center  training.  Outside  of  the 
laboratory,  however,  real-time  simulators  have  been  used  in  the  Air  Force  largely  for  individual 
skills  training  (Bell  &  Waag,  1998).  Due  to  the  peacetime  training  constraints  plus  increased 
operations  and  personnel  tempo,  the  Air  Force  has  undertaken  a  program  of  placing  DMT  assets 
around  the  world  linked  into  a  common  synthetic  battlespace  (Hawley,  1997,  1998).  An 
objective  of  AFRL/HEA’s  DMT  research  program  is  to  identify  the  systems  and  capabilities 
necessary  to  support  these  training  goals. 

Measurement  of  Team  Performance 


Air  Combat  Command’s  (ACC)  concept  for  DMT  is  to  provide  both  team  skills  training 
and  interteam  skills  training.  Team  skills  are  defined  as,  “the  collective  skills  needed  to  perform 
missions...  Teams  are  usually  composed  of  a  single  type  of  weapons  system  from  the  same 
unit.”  Interteam  skills  are,  “  the  composite  skills  required  to  perform  missions,”  where  inter¬ 
team  relationships,  “are  based  on  temporary  assignment  of  teams  from  several  units  to  a  single 
composite  force  package,”  (ACC,  1998,  p.  6).  Currently,  training  requirements  for  these  skills 
are  defined  by  ACC’s  Ready  Aircrew  Program  (RAP)  which  specifies  the  minimum  number  of 
training  sorties  and  events  that  must  be  accomplished  by  each  pilot  to  maintain  combat  mission- 
ready  status.  Proficiency  for  individual  pilots  is  assessed  by  an  instructor  or  flight  leader, 
however,  there  are  few  metrics  for  assessing  proficiency  in  team  or  interteam  skills.  An  objective 
of  AFRL/HEA’s  DMT  research  program  is  to  assist  ACC’s  implementation  of  DMT  by 
developing  metrics  for  assessing  team  performance  and  for  relating  team  skills  to  measures  of 
mission  accomplishment.  RoadRunner  '98  incorporated  candidate  measures  of  team 
performance. 
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Objectives  of  RoadRunner  '98  Exercise 


The  specific  objectives  for  RoadRunner  '98  were  to: 

Demonstrate  DMT  capabilities  necessary  for  combat  mission  training.  The  first  objective 
of  the  RoadRunner  '98  exercise  was  to  create  a  synthetic  battlespace  using  state-of-the-art 
systems  and  to  assess  each  system’s  capability  to  support  DMT.  To  achieve  this  objective, 
warfighters  compared  their  experience  in  RoadRunner  '98  to  their  previous  experiences  and 
identified  DMT  systems  that  successfully  supported  mission  accomplishment,  systems  that 
detracted  from  mission  accomplishment,  and  systems  that  demonstrated  only  limited  success. 

Validate  a  research  approach  for  DMT  training  effectiveness  studies.  The  second 
objective  of  RoadRunner  '98  was  to  support  future  DMT  research  efforts.  To  achieve  this 
objective,  squadron  instructors  and  other  subject-matter  experts  evaluated  team  performance 
using  grading  forms  designed  to  assess  team  performance.  Scores  on  these  fonns,  together  with 
measures  of  mission  accomplishment,  were  analyzed  to  determine  whether  warfighter 
proficiency  was  affected  by  experience  using  DMT  and  to  identify  team  processes  that  could  be 
enhanced  through  DMT. 
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RoadRunner  '98  Systems  and  Approach  to  Training 

RoadRunner  '98  incorporated  geographically  dispersed  warfighter-in-the-loop  (virtual) 
simulations  and  computer-generated  (constructive)  simulations  into  a  virtual  battlespace  using 
DIS  communications  protocols  and  high-bandwidth  data  links.  Following  is  a  brief  description  of 
RoadRunner  '98  systems,  training  scenarios,  and  procedures. 

Systems 

Network  and  locations.  Virtual  simulations  in  RoadRunner  '98  were  located  at  AFRL, 
Mesa  AZ  (F-16  and  A-10),  TACCSF  (F-15  and  virtual  red  air),  and  Tinker  AFB  OK  (AWACS). 
Constructive  simulations  were  located  at  AFRL  (aircraft  plus  surface  vehicles),  TACCSF  (E-3 
AWACS  aircraft  model),  and  AFIWC,  Kelly  AFB  TX  (surface-to-air  missiles  [SAMs]  and 
radars).  In  addition,  real-time  observation  systems  linked  to  the  RoadRunner  '98  network  were 
located  at  the  Theater  Battle  Arena  at  the  Pentagon,  Arlington  VA;  the  Simulation  Analysis 
Facility,  Wright-Patterson  AFB  OH;  and  the  133  ACS,  Ft.  Dodge  IA,  via  the  133  ACS,  Phoenix 
AZ;  see  Figure  1. 


Figure  1.  Map  of  RoadRunner  '98  locations 

All  links  in  the  RoadRunner  '98  network  were  commercial  T-l  lines  except  for  the 
connection  from  TACCSF  to  Tinker  AFB,  which  was  built  around  two,  conventional  telephone 
lines,  and  the  connection  from  AFRL  Mesa,  to  Ft.  Dodge  which  was  an  experimental  satellite 
link.  The  bandwidth  provided  by  T-l  lines  has  proven  to  be  fully  sufficient  in  DMT  exercises 
conducted  prior  to  RoadRunner  '98.  The  telephone-line  connection  to  Tinker  AFB  provided 
significantly  less  bandwidth.  The  data  stream  was  therefore  filtered  down  to  only  voice 
communication  and  aircraft  location.  While  this  bandwidth  was  sufficient  to  support  virtual 


5 


AW  ACS  participation,  the  connection  suffered  from  reliability  problems.  The  satellite  link  was 
adequate  for  observation  of  RoadRunner  '98  missions. 

Cockpits  and  visual  displays.  One  of  the  virtual  cockpits  at  AFRL  Mesa,  was  an  A- 10 
and  four  were  F-16C,  Block  30  MTTs  that  were  developed  by  AFRL  for  the  Air  Force  Reserve 
(Boyle  &  Edwards,  1991);  see  Figure  2a.  These  cockpits  were  equipped  with  AFRL’s  Mobile 
Modular  Display  for  Advanced  Research  and  Training  (M2DART)  which  is  a  full-field  of  view, 
rear-projection,  dome  display  system  (Best,  Wight,  &  Peppier,  1999);  see  Figure  2b.  The 
M2DARTs  provided  out-the-window  visual  imagery  combined  with  the  aircraft’s  head-up 
display  (HUD). 


Figure  2.  a)  F-l 6  MTT  cockpit  b)  M2DART 


Out-the-window,  visual  imagery  was  provided  to  two  of  the  F-l 6  cockpits  from 
Lockheed-Martin  SE2000+  Image  Generators  (IGs)  which  utilized  polygonized  terrain  and 
feature  representations  augmented  with  cell  texture  maps;  see  Figure  3a.  Imagery  was  provided 
to  the  other  two  cockpits  from  Silicon  Graphics,  Inc.  (SGI)  Reality  Monster  IGs  which  used 
polygonized  terrain  representations  augmented  with  geo-specific,  photo-texture  maps.  Three- 
dimensional  feature  models  were  inserted  into  the  SGI  database  only  in  airfield  and  target  areas; 
see  Figure  3b.  Compared  to  the  SGIs,  the  SE2000+  IGs  provided  more  vertical  features  which 
can  be  discerned  at  low  altitude  while  the  SGIs  provided  more  scene  realism  when  the  pilot  was 
flying  over  2000  feet  above  ground  level. 

The  M2DART  provides  an  instantaneous  field-of-view  greater  than  180°  with  high 
brightness  and  contrast  and  a  360°  total  field-of-regard  using  a  head-tracking  system.  Resolution 
is  approximately  4  arc-minutes  per  pixel.  Evaluation  studies  have  demonstrated  that  this 
resolution  does  not  allow  a  pilot  to  detect,  identify,  or  determine  the  aspect  of  other  aircraft  at 
realistic  tactical  ranges.  For  example,  a  common  tactical  formation  during  ingress  would  be  line 
abreast  with  aircraft  separated  by  approximately  one  nautical  mile  (1  NM  =  6,076  feet).  At  this 
distance,  an  F-16’s  longest  dimension  (length)  subtends  less  than  eight  pixels  and  the  shortest 
dimension  (tail  height)  subtends  less  than  three.  Against  a  cluttered  background,  a  target  model 
of  this  size  is  very  difficult  to  detect.  To  increase  target  visibility,  aircraft  models  in  the 
M2DART  are  replaced  by  a  point  light  when  range  exceeds  4,000  feet.  This  light  is  colored  blue 
or  red  to  indicate  friendly  or  foe  between  4,000  feet  and  3  NM  and  white  from  3  NM  to  7  NM. 
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While  this  system  increases  detection  and  identification  ranges  to  acceptable  distances,  pilots 
still  cannot  determine  aspect  beyond  4,000  feet. 


Figure  3.  a)  SE2000+  sample  image  b)  SGI  sample  image 


The  A- 10  at  AFRL/HEA  was  equipped  with  an  earlier  model  DART  visual  display.  This 
DART  did  not  provide  out-the-window  visual  at  the  aircraft’s  6  o'clock  position.  Imagery  was 
supplied  from  a  Lockheed-Martin  SE  2000. 

The  four,  virtual  F-15  cockpits  at  TACCSF  were  WTTs.  The  visual  displays  on  these 
cockpits  consisted  of  a  single  CRT  that  incorporated  forward,  out-the-window  visual  imagery 
and  the  aircraft’s  HUD;  see  Figure  4a.  In  addition  to  the  virtual  F-15s,  two  virtual  MiG-29  red 
air  stations  were  located  at  TACCSF  supported  by  a  red  air  controller;  see  Figures  4b  and  c. 
These  stations  incorporated  aircraft  instruments  and  HUD  with  only  a  limited  visual  display.  The 
virtual  AWACS  stations  were  full  mission  trainers  located  at  the  AW  ACS  training  facility, 
Tinker  AFB  OK,  and  adapted  for  participation  in  DMT  exercises  (see  Figure  4d). 

Constructive  forces.  Incorporating  computer-generated,  constructive  forces  into  a  DMT 
exercise  allows  virtual  players  to  interact  with  many  friendly,  neutral,  and  opposing  players 
without  requiring  real-time  participation  of  human-in-the-loop  simulators.  The  limitations  of 
constructive  forces  are  that  each  constructive  model  has  different  strengths  and  capabilities  and 
that  each  must  be  preprogrammed  for  each  scenario.  In  addition  to  the  constructive  AWACS 
aircraft  model,  the  systems  used  in  RoadRunner  '98  were  AFRL/HEA ’s  Automated  Threat 
Engagement  System  (ATES),  TacAirSOAR,  and  AFIWC’s  Integrated  Air  Defense  System 
(IADS)  (Cicero,  1998). 

The  ATES  constructive  forces  simulation  was  developed  at  AFRL/HEA  to  support 
research  on  multiship  training  systems  (Rogers,  1992;  Platt  &  Crane,  1993).  ATES  is  a  blend  of 
several  programs  from  various  government  agencies  and  a  commercial  vendor  that  can  generate 
synthetic  aircraft,  ground  vehicles,  and  surface-to-air  threats.  During  RoadRunner  '98,  ATES 
provided  enemy  fighter  aircraft,  a  KC-135  aerial  refueling  tanker,  and  ground  vehicles.  ATES 
aircraft  models  are  both  scriptable  and  autonomous.  Fighter  aircraft,  for  example,  can  be 
scripted  to  assume  a  combat  air  patrol  (CAP)  where  they  will  remain  throughout  a  scenario  until 
either  a  preset  length  of  time  has  passed  or  until  an  opposing  aircraft  comes  within  a  preset 
range.  The  ATES  fighters  will  then  become  autonomous  and  attack  using  an  intelligent  flight 
model  that  incorporates  knowledge  of  weapons  and  tactics  appropriate  for  the  aircraft  type. 
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ATES  fighters  used  during  RoadRunner  '98  were  MiG-29s  and  Su-27s  armed  with  AA-lOa 
missiles,  and  MiG-23  attack  aircraft. 


Figure  4.  c)  Red  controller  d)  AW  ACS  simulator 


TacAirSOAR  was  developed  at  the  University  of  Michigan  and  the  Information  Sciences 
Institute  of  the  University  of  Southern  California  under  the  Defense  Advanced  Research  Projects 
Agency's  Synthetic  Theater  of  War  program  (Jones,  Laird,  &  Neilson,  1998).  The  goal  of 
TacAirSOAR  is  to  develop  human-like  synthetic  entities  for  populating  simulation 
environments.  It  accomplishes  this  by  using  SOAR  architecture  to  integrate  a  wide  range  of 
intelligent  capabilities,  including  real-time  hierarchical  execution  of  complex  goals  and  plans, 
communication  and  coordination  with  humans  and  simulated  entities,  maintenance  of  situational 
awareness,  and  ability  to  accept  new  orders  while  in  flight.  The  primary  role  of  TacAirSOAR 
controlled  aircraft  in  RoadRunner  '98  was  to  enhance  the  scenario  by  providing  realistic  forces 
flying  in  strike  packages  with  other  computer-generated  forces  and  with  manned  simulators. 
TacAirSOAR  aircraft  used  during  RoadRunner  '98  F-16C  Block  50s  armed  with  High-speed 
Anti-Radiation  Missiles  (HARMs),  F-16C  Block  30  strikers,  and  helicopters. 
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AFIWC’s  IADS  model  provided  a  basic  air  defense  system  consisting  of  two  early 
warning  radars  and  several  SAM  batteries.  The  SAM  systems  were  one  SA-2,  two  SA-6,  and 
three  SA-8  sites.  Command  and  control  functions  were  simulated  through  interactions  between 
the  early  warning  radars  and  SAM  sites. 

Control  consoles.  The  simulation  control  consoles  at  the  various  RoadRunner  '98  sites 
provided  both  an  interface  for  system  operators  to  conduct  a  mission  and  an  observation  station 
for  instructors  to  monitor  team  performance. 

At  AFRL/HEA,  the  control  console  incorporated  an  operator’s  station,  the  exercise 
director’s  station,  and  an  observer’s  station  for  the  F-16  instructor  pilots;  See  Figure  5a.  The 
instructor  monitored  mission  progress  on  a  six-screen,  video  display  system  which  showed  the 
forward,  out-the-window  view  from  each  of  the  cockpits  together  with  a  plan-view  or  map-like 
display  of  the  gaming  area  showing  icons  for  each  entity  in  the  scenario.  The  sixth  video  display 
showed  the  radar  screens  from  each  of  the  four  cockpits;  see  Figure  5b.  The  exercise  director  at 
AFRL/HEA  could  communicate  with  all  participants  and  issue  global  freeze/unfreeze 
commands.  The  exercise  director  could  also  communicate  privately  with  the  control  console 
operators  at  the  other  RoadRunner  '98  sites  either  through  a  digital  voice  channel  or  using  a 
telephone  for  backup. 


F-16  #1 
forward  view 

F-16  #1 
radar 

F-16  #3 
radar 

F-16  #3 
forward  view 

F-16  #2 
radar 

F-16  #4 
radar 

F-16  #2 
forward  view 

Plan  view 
display 

F-16  #4 
forward  view 

Control  capabilities  at  TACCSF  incorporated  repeater  screens  for  AW  ACS,  F-15Cs,  and 
virtual  MiGs  together  with  a  stealth  view  of  the  battlespace.  Data  filtering  and  the 
communications  link  with  AW  ACS  at  Tinker  AFB  were  also  located  at  TACCSF.  Control 
capabilities  at  Tinker  AFB  were  limited  to  communications  systems  with  TACCSF  and  local 
instructor-operator  stations. 

Debrief  systems.  Previous  research  (Bell  &  Waag,  1998;  Berger  &  Crane,  1993;  Houck 
et  al,  1991)  has  demonstrated  that  value  of  mission  replay  in  supporting  team  debrief.  The 
debrief  system  at  AFRL/HEA  supported  synchronized  replay  of  a  plan-view  display  together 
with  information  from  each  cockpit  (radar  screen,  HUD,  Radar  Warning  Receiver  [RWR],  and 
Stores  Management  System[SMS]);  see  Figures  6  a  and  b.  Debrief  systems  at  TACCSF  for  the 
F-15  pilots  and  at  Tinker  for  AW  ACS  teams  provided  only  the  plan-view  which  was  presented  in 
map-like  format  at  TACCSF  and  as  an  AW ACS  display  at  Tinker. 
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Figure  6.  a)  AFRL  debrief  system  b)  Diagram  of  AFRL  debrief  screens 

Training  Scenarios 

The  training  scenarios  used  in  RoadRunner  '98  were  developed  by  an  exercise  planning 
team  (white  cell)  that  included  subject-matter  experts  from  AFRL/HEA  and  TACCSF  together 
with  intelligence  and  training  officers  from  participating  units.  Since  all  participants  in 
RoadRunner  '98  were  mission-qualified  warfighters,  it  was  not  necessary  to  train  basic  skills  or 
procedures.  Rather,  mission  scenarios  were  designed  to  enhance  skills  that  previous  research 
suggested  would  benefit  from  the  unique  capabilities  of  DMT  compared  to  single-ship 
simulation  or  large-force  exercises.  In  single-ship  simulator  training  exercises  such  as  responding 
to  in-flight  emergencies,  an  instructor  introduces  an  emergency  giving  the  student  the 
opportunity  to  respond.  Events  are  highly  scripted  and  the  instructor  can  easily  evaluate  good  vs 
poor  performance.  In  contrast,  large-force  exercises  are  much  less  scripted  at  the  level  of 
individual  warfighters.  Evaluators  will  know  where  and  when  forces  will  engage  but  will  have 
only  limited  control  over  each  participant’s  experience.  DMT  incorporates  the  scenario  control 
qualities  of  single-ship  simulator  training  with  capabilities  to  enhance  team  and  inter-team  skills. 
The  actions  of  computer-generated  forces  in  RoadRunner  '98  were  designed  to  create  training 
scenarios  that  would  exercise  specific  individual,  team,  and  inter-team  skills.  Among  fighter 
pilots,  these  can  be  high-level,  individual  skills  such  as  using  air-to-air  radar  to  build  situation 
awareness  or  react  to  threats;  team  skills  such  as  communication  and  maintaining  mutual 
support;  or  inter-team  skills,  which  require  coordination  among  pilots  and  air  weapons 
controllers  or  a  FAC.  RoadRunner  '98  scenarios  were,  therefore,  generated  based  on  exercise 
objectives,  technical  feasibility,  similarity  to  missions  in  the  F-16  and  F-15  RAP,  and  integration 
of  mission  skills  that  could  be  enhanced  by  experience  using  DMT. 

Gaming  area.  RoadRunner  '98  exercises  were  conducted  over  a  synthetic  Red  Flag 
gaming  area.  This  area  was  selected  because  it  was  familiar  to  participants  and  databases  were 
available  at  all  of  the  RoadRunner  '98  sites.  For  all  missions,  the  eastern  side  of  the  gaming  area 
was  friendly  (blue)  controlled  territory  while  the  western  side  was  enemy  (red)  controlled. 
Missions  began  with  the  blue  forces  airborne  just  to  the  east  of  a  landmark  known  as  Student 
Gap;  see  Figure  7,  point  1.  Except  for  the  CAS  and  Defensive  Counterair  (DCA)  missions,  blue 
forces  departed  from  friendly  airspace  (pushed)  at  Student  Gap  and  proceeded  west  crossing  the 
Forward  Edge  of  the  Battle  Area  (FEBA)  and  attacking  surface  targets  in  enemy  territory.  For 
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the  CAS  mission,  targets  were  surface  vehicles  near  the  FEBA  and  for  DCA  both  F-15s  and  F- 
16s  defended  blue  airspace  against  incoming  red  aircraft. 

Red  airspace  was  defended  by  SAMs  and  Anti-Aircraft  Artillery  (AAA)  positioned  near 
the  FEBA.  SAM  sites  are  depicted  in  Figure  7  by  hexagons  labeled  2,  6,  and  8.  Enemy 
controlled  territory  was  also  defended  by  constructive  Su-27  fighters  plus  constructive  and 
virtual  MiG-29s  stationed  at  Tolicha  and  Tonopah  airfields  near  the  western  edge  of  the  gaming 
area.  RoadRunner  '98  scenarios  took  place  within  a  gaming  area  approximately  90  by  120  NM. 


Figure  7.  Map  of  SAT  1  mission 

Unconstrained  environment.  In  a  normal,  range  exercise  conducted  at  Red  Flag,  safety, 
security,  and  environmental  considerations  impose  limits  on  where  aircraft  may  fly,  what 
altitudes  they  may  select,  and  where  they  may  deliver  ordnance.  Warfighter  training  in 
synthetic  battlespace  is  not  constrained  by  these  limitations.  RoadRunner  '98  mission  scenarios 
developed  by  the  white  cell  required  pilots  to  fly  specified  routes  to  and  from  the  target  areas. 
These  routes  were  selected  to  insure  that  pilots  would  encounter  constructive  entities  positioned 
on  the  ground  or  programmed  to  fly  through  the  gaming  area.  If  attacked,  however,  pilots  were 
free  to  deviate  from  planned  routes,  altitudes,  or  airspeeds.  Tactical  execution  in  RoadRunner 
'98  could  be  based  entirely  on  mission  requirements  and  not  limited  by  artificial  constraints. 

The  virtual  F-16s  at  AFRL/HEA  had  the  option  to  select  real-time  kill  removal  (shields- 
down)  or  to  disable  kill-removal  (shields-up).  When  flying  with  shields-down,  a  virtual  F-16 
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that  was  hit  by  AAA  or  a  missile  would  freeze-up  and  be  removed  from  the  battlespace.  When 
flying  shields-up,  the  out-the-window  visual  display  flashed  red  for  several  seconds  when  the 
aircraft  was  hit  but  the  pilot  kept  flying  and  was  able  to  complete  the  mission.  The  number  of 
times  that  each  F-16  was  killed  was  recorded.  All  missions  except  for  Interdiction  were  flown 
shields-up.  This  option  was  selected  so  that  pilots  would  be  able  to  evaluate  all  aspects  of 
training  in  the  virtual  battlespace.  The  F-15  cockpits  did  not  have  the  capability  to  fly  shields-up 
but  could  be  regenerated  quickly  after  being  shot  down. 

Roles  of  teams  and  mission  commander.  For  all  RoadRunner  '98  missions  except  CAS 
and  DCA,  the  mission  of  the  virtual  F-15s  was  to  conduct  a  pre-strike  sweep  to  eliminate  red  air 
threats  by  pushing  approximately  five  minutes  ahead  of  the  F-16  strike  package  and  engaging  red 
fighters.  During  CAS  and  DCA,  the  F-15s  defended  an  assigned  area  against  incoming  red 
fighters. 

The  role  of  the  virtual  F-16s  in  RoadRunner  '98  missions,  except  for  DCA,  was  air-to- 
surface  attack  using  non-precision  munitions  such  as  Mk-82,  Mk-84,  and  CBU-87  bombs.  The 
targets  and  types  of  enemy  defenses  varied  across  missions.  In  the  DCA  mission,  the  virtual  F- 
16s  defended  an  assigned  area  against  incoming  red  fighters. 

The  task  of  the  AW  ACS  team  was  to  provide  information  necessary  to  conduct  counter¬ 
force  maneuvers.  This  would  include  detecting  and  tracking  all  aircraft  and  identification  as  to 
whether  a  beyond  visual  range  (BVR)  target  aircraft  was  friendly  or  hostile  as  defined  by  the 
rules  of  engagement.  AWACS  Weapons  Directors  would  communicate  this  information  to  pilots 
using  a  standardized  set  of  brevity  codes  to  describe  the  number  of  enemy  threats,  their  locations, 
heading,  altitudes,  airspeeds,  and  maneuvers.  Weapons  directors  also  provided  vectors  to  threat 
and  friendly  aircraft  as  requested  by  pilots. 

The  lead  F-16  pilot  was  the  Mission  Commander  for  all  RoadRunner  '98  missions  except 
for  DCA  where  the  lead  F-15  pilot  was  commander.  The  commander’s  task  was  to  coordinate 
the  actions  of  the  different  virtual  teams  within  the  constraints  of  the  Air  Tasking  Order  (ATO), 
brief  the  teams  on  the  mission  plan,  and  to  lead  debrief.  During  RoadRunner  '98,  the  Mission 
Commander  had  no  control  over  constructive,  blue  elements. 

Composite  Force  Missions 

Six  composite  force  missions  were  selected  for  RoadRunner  '98.  The  criteria  for 
selection  was  that  each  mission  required  at  least  one  four-ship  formation  of  both  F-15s  and  F-16s 
and  that  the  mission  was  part  of  the  current  RAP  tasking.  For  each  mission,  white  cell 
intelligence  officers  provided  teams  with  an  ATO,  special  instructions  (SPINS),  route  of  flight, 
target,  and  known  threats  (see  Appendix  A).  Instructors  and  evaluators  were  also  provided  with  a 
chart  showing  red  force  actions  that  had  been  scripted  for  each  mission;  Figure  7  is  an  example 
of  this  chart.  Missions  were  flown  according  to  the  schedule  described  in  Table  la.  All  missions 
began  with  blue  forces  initialized  in  flight  east  of  Student  Gap  with  the  F-16s  in  formation 
behind  a  constructive  KC-135  tanker  as  if  they  had  just  completed  refueling.  The  F-15s  were  in 
an  orbit  nearby.  The  actual  push  from  Student  Gap  occurred  approximately  ten  minutes  after  the 
mission  began  to  allow  the  AWACS  crew  to  identify  aircraft  on  their  radar  displays  and  to  build 
situation  awareness  of  threat  activity.  Missions  were  tenninated  by  the  exercise  director  when 
the  virtual  F-15s  and  F-16s  returned  to  blue  airspace.  This  required  30-45  minutes  depending  on 
the  mission. 
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Table  la.  Schedule  for  RoadRunner  '98  week 


Mon 

13  Jul 

Tue 

14  Jul 

■ 

I 

Thu 

16  Jul 

Fri 

17  Jul 

Travel, 

Inbrief, 

Questionnaires 

Test  1: 

Surface  Attack 
Tactics 

Interdiction 

Suppression  of 
Enemy  Air 
Defense: 
Conventional 

Test  2: 

Surface  Attack 
Tactics; 
Outbrief, 
Questionnaires 

Familiarization  ride 

Offensive  Counter 
Air: 

Air-to-Ground 

Defensive  Counter 
Air 

Close  Air  Support 

Familiarization.  On  Monday  afternoon  pilots  flew  a  familiarization  mission.  Pilots  were 
initialized  over  Nellis  AFB  and  flew  around  the  local  area  using  their  radars  and  missiles  to 
engage  each  other.  The  purpose  of  this  exercise  was  to  become  familiar  with  visual  appearance 
of  other  aircraft  and  to  observe  the  red,  flashing  screens  indicating  that  the  pilot  had  been  killed. 
Pilots  were  re-initialized  over  Student  Gap  and  flew  air-to-air  sweeps  against  constructive  forces 
with  AW  ACS  support.  Participants  used  their  respective  debrief  systems  to  replay  the  mission. 

Surface  Attack  Tactics  IS  AT  1  and  21 — Pre-test  and  Post-test  missions.  After  the 
familiarization  ride,  the  first  (Tuesday  morning)  and  last  (Friday  morning)  RoadRunner  '98 
missions  flown  were  SAT  sorties  which  are  air-to-surface,  tactical  strike  missions  in  a  high 
threat  environment.  In  SAT  1  and  2,  the  F-15s  were  tasked  to  provide  a  pre-strike  sweep  while 
the  virtual  F-16s  were  tasked  to  employ  ordnance  against  a  fuel  pump-house  and  a 
communications  facility  located  at  Tolicha  airfield  indicated  by  a  triangle  on  Figure  7.  In 
addition  to  the  virtual  simulators,  two  flights  of  constructive  F-16s  also  participated.  The  first 
was  a  four-ship  formation  of  F-16  Block  50  aircraft  equipped  with  HARMs  that  were  to  be 
employed  against  two  SA-6  sites.  The  second  constructive  element  was  a  four-ship  fonnation  of 
F-16  Block  30  aircraft  that  was  also  tasked  to  attack  Tolicha  airfield  1.5  minutes  following  the 
virtual  aircraft.  Virtual  MiG-29s  engaged  the  F-15s  during  their  pre-strike  sweep  and  were  re¬ 
initialized  to  engage  the  F-16s  during  their  egress.  The  virtual  F-15s  and  F-16s  also  encountered 
both  blue  and  red  helicopters  during  their  ingress.  SAT  missions  1  and  2  were  nearly  identical 
except  for  the  route  of  flight  and  initial  locations  of  red  forces.  SAT  1  and  SAT  2  served  as  pre¬ 
test  and  post-test  missions  designed  to  be  relatively  high-intensity  missions  that  would  present 
significant  challenges  to  participating  pilots  and  AW  ACS. 

Offensive  Counterair:  Air-to-Ground  (OCA  A-G).  The  OCA  A-G  mission  was  similar  to 
SAT  but  with  fewer  challenging  elements.  Virtual  F-15s  and  F-16s  pushed  from  a  point  north  of 
Student  Gap  while  the  constructive  flights  of  F-16C  Block  50s  and  Block  30s  pushed  from 
another  point.  Constructive  Block  50s  were  tasked  to  suppress  SA-6  sites  with  HARMs  and  the 
constructive  Block  30s  employed  ordnance  on  a  second  airfield.  The  targets  were  larger  than  for 
the  SAT  mission,  e.g.,  runway  intersections,  there  were  no  helicopters  in  the  scenario,  and  there 
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were  fewer  air  and  surface  threats.  However,  an  unbriefed  SA-6  site  went  active  while  the  virtual 
F-16s  were  egressing. 


Table  lb.  Schedule  for  Tue,  14  Jul  98 


Time 

(DC) 

TBA 

Time 

(OK) 

Tinker 

Time 

(NM) 

TACCSF 

Time 

(AZ) 

AFRL 

Team  A 

Team  B 

Team  C 

1000- 

1030 

0900- 

0930 

0800- 

0830 

0700- 

0730 

Brief 

Test  Mission 

Brief 

Test  Mission 

1030- 

1100 

0930- 

1000 

0830- 

0900 

0730- 

0800 

1100- 

1130 

1000- 

1100 

0900- 

0930 

0800- 

0830 

Fly 

Testl 

1130- 
i  1200 

1030- 
1100  £ 

0930-  • 
1000 

0830- 

0900 

Brief 

Test  Mission 

1200- 

1230 

1100- 

1130 

1000- 

1030 

0900- 

0930 

Debrief 

Test  Mission 

1230- 
l  1300 

1130- 

1200 

1030- 

1100 

0930- 

1000 

:  • 

Fly 

Test  2 

1300- 

1330 

1200- 

1230 

1100- 

1130 

1000- 

1030 

1330- 

1400 

1230- 

1300 

1130- 

1200 

1030- 

1100 

Lunch 

break 

Debrief 

Test  Mission 

1400- 

1430 

1300- 
::  1330 

1200- 

1230 

1100- 

1130 

Fly 

Test  1 

S  1430- 
1500 

,  1330- 
1400 

1230- 

1300 

1130- 

1200 

Brief 

OCAA-G 

1500“ 

1530 

1400- 

1430 

1300- 

1330 

1200- 

1230 

Lunch 

break 

Debrief 

Test  Mission 

1530- 

1600 

1430- 

1500 

■:?  1330-  . 
1400 

1230- 

1300 

Fiy 

OCAA-G 

1600- 

1630 

1500- 

1530 

1400- 

1430 

1300- 

1330 

Brief 

OCAA-G 

1630- 

1700 

1530- 

1600 

1430- 

1500 

1330- 

1400 

Debrief 

OCAA-G 

Lunch 

break 

1700- 
1730  f 

1600- 

1630 

1500- 

1530 

1400- 

1430 

Fly 

OCAA-G 

1730- 
W  1800  ■ 

1630- 

1700 

•  1530-  ! 

1600 

1430- 

1500 

Brief 

OCAA-G 

1800- 

1830 

1700- 

1730 

1600- 

1630 

1500- 

1530 

Debrief 

OCAA-G 

I.  1830- 
1900 

1730- 

1800 

1630- 
1700  ' 

1530-  H 
1600 

Fly 

OCAA-G 

1900- 

1930: 

1800- 

1830 

1700- 

,1730 

1600- 

1630 

1930- 

2000 

1830- 

1900 

1730- 

1800 

1630- 

1700 

Debrief  OCA 
A-G 

2000- 

2030 

1900- 

1930 

1800- 

1830 

1700- 

1730 

2030- 

2100 

1930- 

2000 

1830- 

1900 

1730- 

1800 
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Interdiction.  The  Interdiction  mission  incorporated  more  air  and  surface  threats  than 
OCA  A-G  and  was  flown  shields-down.  The  target  was  a  storage  depot.  Constructive  Block  50s 
were  tasked  to  suppress  an  SA-6  site  and  an  SA-2  site  near  the  target.  Constructive  Block  30s 
attacked  an  airfield  due  south  of  the  storage  depot.  Egress  routing  was  identical  for  virtual  and 
constructive  players. 

Defensive  Counterair  (DCA).  This  was  the  only  exclusively  air-to-air  mission  during 
RoadRunner  '98.  Virtual  F-15s  were  tasked  to  defend  a  corridor  north  of  a  specified  latitude 
while  F-16s  defended  a  corridor  to  the  south.  Each  flight  assumed  a  Combat  Air  Patrol  (CAP) 
staying  west  of  Student  Gap  but  east  of  the  FEBA  to  avoid  red  surface  threats.  Both  flights 
defended  blue  territory  from  air  attacks  by  constructive  MiG-23  strikers  that  were  escorted  by 
virtual  and  constructive  MiG-29s  plus  constructive  Su-27s.  The  DCA  mission  was  longer  than 
the  other  RoadRunner  '98  missions  requiring  at  least  45  minutes  to  complete. 

Suppression  of  Enemy  Air  Defenses — Conventional  (SEAD-C).  In  the  SEAD-C  mission, 
the  virtual  F-16s  were  tasked  to  deliver  non-precision  guided  munitions  against  a  non-active  SA- 
2  site  while  constructive  F-16  flights  attacked  active  SA-6  sites  and  two  airfields.  In  addition  to 
the  virtual  and  constructive  air  threats  seen  on  previous  RoadRunner  '98  missions,  a  constructive 
flight  of  MiG-23  strike  aircraft  flew  through  the  area  to  attack  blue  territory.  Virtual  F-15s  were 
tasked  during  the  mission  to  swing  from  pre-strike  sweep  to  a  DCA  role  and  engage  the  MiG- 
238. 

Close  Air  Support.  For  the  CAS  mission,  the  F-15s  flew  a  DCA  role  while  F-16s 
provided  CAS  for  blue  surface  forces  near  the  FEBA.  This  was  a  low-threat  mission  in  that  there 
was  an  SA-8  site  near  the  target  area  but  no  air  threats.  F-16  attacks  were  directed  by  an  A-10 
pilot  serving  as  an  airborne  FAC.  Targets  were  formations  of  T-72  tanks. 

Schedule  and  Daily  Operations 

Three  teams  of  pilots  and  AW  ACS  crews  participated  in  RoadRunner  '98  which  required 
that  each  mission  be  conducted  three  times.  To  accomplish  this  and  to  adhere  to  the  overall 
schedule  listed  in  Table  la,  a  daily  schedule  was  adopted  that  overlapped  briefings,  missions, 
and  debriefings;  see  Table  lb.  At  the  beginning  of  each  briefing  period,  the  Mission 
Commander  telephoned  the  other  flight  lead  and  the  AWACS  senior  controller  to  review  the 
mission  plan.  Each  team  then  briefed  their  plan  until  their  assigned  simulator  time.  Although 
scenarios  were  programmed  to  last  less  than  45  minutes,  each  team  was  allotted  60  minutes  of 
simulator  time  in  case  of  technical  problems.  When  the  scenario  was  completed  without 
difficulty  and  there  was  time  available,  the  F-16  flight  at  AFRL/HEA  went  offline  and  conducted 
4  vs  X  air-to-air  engagements  against  a  constructive  red  force.  See  Appendix  B  for  descriptions 
of  sample  engagements. 

Teams  conducted  their  debriefing  in  two  phases:  local  (team)  and  wide-area  (inter-team). 
During  local  debrief,  teams  would  replay  the  mission  tape  with  the  goal  of  understanding  what 
happened  during  the  mission  and  generating  points  for  review  by  all  participants.  After  30 
minutes,  teams  were  to  prepare  a  list  of  topics  (high  points  and  low  points)  and  fax  them  to  the 
Mission  Commander.  The  Mission  Commander  then  had  ten  minutes  to  prepare  the  inter-team 
debrief.  Two  secure  speaker  phones  were  available  at  AFRL/HEA.  To  begin  the  debrief,  the  F- 
16  team  was  to  call  TACCSF  and  Tinker  to  establish  secure  voice  communications.  All  teams 
would  then  restart  their  tapes  at  the  beginning  of  the  mission  and  pause  for  discussion  as  directed 
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by  the  commander.  At  the  conclusion  of  debrief,  participants  filled  out  a  brief  questionnaire  that 
focused  on  identifying  systems  or  procedures  that  reduced  training  value  of  the  mission. 

Personnel 


The  objectives  of  the  RoadRunner  '98  exercise  focused  on  determining  the  effectiveness 
of  DMT  systems  for  training  operational  warfighters.  Participants  were  therefore  pilots  and 
AW  ACS  mission  crews  assigned  to  operational  squadrons  and  represented  a  mix  of  experience 
levels. 

F-16  pilots.  Fifteen  F-16  pilots  participated  in  this  exercise.  The  522nd  Fighter  Squadron 
(FS)  and  523  FS  from  the  27  FW,  Cannon  AFB  NM;  the  Iowa  Air  National  Guard;  and  the  185 
and  132  FWs,  each  sent  a  team  of  five  pilots.  Squadrons  were  requested  to  send  one  instructor 
pilot,  one  mission  commander,  one  2-  or  4-ship  element  leader,  and  two  mission-ready  wingmen. 
The  mean  number  of  F-16  flight  hours  for  these  pilots  is  shown  in  Figure  8a.  Five  of  the 
wingmen  plus  one  element  leader  had  no  flight  experience  other  than  F-16  and  undergraduate 
pilot  training.  The  other  two  wingmen  had  over  1,000  hours  each  in  previous  assigmnents  to 
non-fighter  aircraft.  The  four  mission  commanders  and  three  instructors  had  between  700  and 
1,600  hours  experience  in  F-16s.  Three  of  these  pilots  had  over  1,000  hours  experience  in 
nonfighter  aircraft.  For  further  analyses,  pilots  were  grouped  into  two  categories,  less 
experienced  pilots  (wingmen  and  element  leaders)  and  more  experienced  (mission  commanders 
and  instructors);  see  Figure  8b. 
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Figure  8.  a)  F-16  flight  hours  by  squadron 
qualification 


Experience 

b)  F-16  flight  hours  for  more  and  less 
experienced  pilots 


F-15  pilots.  Six  F-15  pilots  participated  in  RoadRunner  '98:  five  from  the  33  FW,  Eglin 
AFB  FL  and  one  from  the  United  States  Air  Forces  in  Europe  who  was  currently  assigned  to  a 
staff  position.  These  pilots  averaged  over  1,000  F-15  hours  experience  with  only  one  having 
fewer  than  500  hours. 

AW  ACS  teams.  Fifteen  AW  ACS  mission  crewmembers  from  the  552  ACW,  Tinker 
AFB  OK  divided  into  three  teams  participated  in  RoadRunner  '98.  Each  team  included  one 
instructor,  two  weapons  directors,  and  two  air  surveillance  technicians.  Mean  experience  level 
was  620  flight  hours  in  the  E-3  and  201  hours  in  the  AWACS  simulator. 
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Other  participants.  Two  A-10  instructor  pilots  from  the  355th  Wing,  Davis-Monthan 
AFB  AZ,  participated  as  airborne  FACs  during  the  CAS  mission.  Both  of  these  pilots  had  over 
1,000  hours  experience  in  the  A-10.  Two  pilots  from  the  549th  Combat  Training  Squadron  (Air 
Warrior),  Adversary  Tactics  Flight,  at  Nellis  AFB  NV,  participated  as  virtual  red  air  threats  at 
TACCSF;  and  three  intelligence  officers,  one  from  each  F-16  team,  prepared  briefing  materials 
and  briefed  their  pilots  at  AFRL/HEA. 

Data  Collection 

Separate  data  collection  procedures  and  instruments  were  used  for  pilots  and  AWACS 
teams.  Data  included  mission  performance  scores,  instructor  or  subject-matter  expert 
evaluations,  participant  questionnaires,  and  participant  comments. 

Pilot  data.  Questionnaires.  After  receiving  an  in-brief  and  overview  of  the  exercise,  all 
pilots  completed  a  questionnaire  asking  them  to  rate  the  effectiveness  of  their  current  aircraft 
training  program  with  respect  to  several  mission  tasks  and  elements.  Effectiveness  of  the  current 
program  was  rated  on  a  zero  to  four  scale  with  the  anchors: 

0  =  Negative  training  for  combat;  aircraft  training  reduces  combat  readiness,  and 

4  =  Total  positive  training;  aircraft  training  is  equivalent  to  combat  experience. 

After  the  final  DMT  mission,  pilots  completed  a  similar  questionnaire  rating  the 
effectiveness  of  DMT  for  training  the  same  list  of  tasks  and  elements.  For  this  questionnaire,  the 
scale  anchors  were: 

0  =  Negative  training  for  combat;  DMT  reduces  combat  readiness,  and 

4  =  Total  positive  training;  DMT  is  equivalent  to  combat  experience. 

Mission  performance  on  SAT  1  and  2.  Measures  of  mission  performance  were  collected 
from  the  DIS  log  files.  F-16  bomb  score  (distance  in  feet  from  the  designated  mean  point  of 
impact)  was  computed  directly  from  recorded  data.  Red  kills  and  blue  losses  to  surface  and  air 
threats  could  not  be  determined  directly  or  automatically  from  data.  When  flying  shields-up,  a 
virtual  F-16  would  not  disappear  from  the  battlespace  when  killed  by  a  red  threat.  Therefore,  a 
constructive  threat  would  continue  to  track  the  F-16  and  kill  it  over  and  over  until  the  exercise 
director  could  order  it  to  disengage.  This  problem  and  other  difficulties  with  constructive  forces 
required  interpretation  by  a  subject-matter  expert.  For  scoring  purposes,  only  the  first  kill  was 
counted. 

Interviews  and  critiques.  Pilot  critiques  were  collected  after  each  mission  and  on  the 
training  effectiveness  questionnaires.  F-16  pilots  were  also  interviewed  after  three  of  four  DMT 
missions  regarding  system  successes  and  shortfalls  and  their  judgments  regarding  best  use  of 
DMT. 

F-16  Instructor  Pilot  evaluation.  F-16  team  process  data  were  collected  for  this  effort 
using  two  different  approaches  that  have  been  successfully  used  in  previous  training  research 
efforts  (see,  Dwyer,  Oser,  Salas,  &  Fowlkes,  1999).  While  both  of  the  approaches  were  designed 
to  assess  performance  and  provide  feedback  to  teams,  the  nature  of  the  type  of  data  obtained 
differ. 
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Table  2.  Dimensions  and  key  factors  for  Teamwork  Oriented  Measurement  (TOM)  scales 


DIMENSIONS  (Higher  order  skills) 

COMMUNICATION  -  the  exchange  of  information  between  two  or  more  team  members. 
Descriptors  - 

(1)  Information  Exchange  -  involves  the  clear,  concise,  and  accurate  exchange  of 
information  between  elements  of  the  team, 

(2)  Information  Clarification  -  involves  the  detection  of  inaccurate  or  incomplete 
information  and  taking  corrective  action, 

(3)  Information  Cadence  -  involves  the  timing,  rhythm,  and  flow  of  information,  and 

(4)  Information  Format  -  involves  the  terminology  and  order  of  the  infonnation 

SITUATION  AWARENESS  -  the  exchange  of  information  used  to  develop  and  maintain  an 
accurate  perception  of  the  operating  environment. 

Descriptors  - 

(1)  Maintaining  an  Overall  Mission  View  -  involves  interactions  involving  mission 
goals  and  how  situational  factors  are  affecting  the  mission  goals, 

(2)  Monitoring  Mission  Deviations-involves  interactions  related  to  detection  and  com¬ 
munication  of  changes  in  operational  environments  that  could  affect  the  mission  plan, 

(3)  Monitoring  Mission  Progress  -  involves  interactions  related  to  the  current  location 
and  status  of  mission  assets, 

(4)  Understanding  Current  Mission  State  -  involves  interactions  that  demonstrates  the 
team’s  shared  and  common  understanding  of  the  mission  status,  and 

(5)  Assessing  Future  Mission  States  -  involves  interactions  related  to  potential 
modification  of  mission  plan 

ADAPTABILITY/FLEXIBILITY  -  exchange  of  information  related  to  modification  of  plans 
Descriptors  - 

(1)  Maintaining  the  Pre-Briefed  Plan  -  involves  interactions  demonstrating  that  the  team 
will  not  modify  its  existing  plan, 

(2)  Changing  to  a  Pre-Briefed  Alternate  Plan  -  involves  interactions  demonstrating  that 
the  team  will  be  modifying  its  current  plan  to  a  previously  briefed  alternate  plan,  and 

(3)  Changing  to  a  Non-Briefed  Plan  -  involves  interactions  demonstrating  the  team  will 
modify  its  current  plan  to  a  plan  that  was  not  previously  briefed 

CREW  COORDINATION  -  the  exchange  of  information  related  to  team  synchronization. 
Descriptors  - 

(1)  Providing  Information  in  Advance  -  involves  interactions  related  to  team  members 
anticipating  team  members'  needs  for  infonnation  and  providing  the  information  prior  to 
its  request, 

(2)  Providing  Back-Up  When  Required  -  involves  interactions  related  to  team  members 
recognizing  need  for  assistance  and  providing  such  assistance,  and 

(3)  Maintaining  Contracts  -  involves  interactions  related  to  team  member  implicit 

interactions  agreed  upon  during  the  mission  brief _ 
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The  first  approach,  the  Teamwork  Observation  Measure  (TOM),  focuses  on  the 
identification  of  strengths  and  weaknesses  at  the  higher  order  skill,  or  dimensional  level.  Four 
dimensions  (or  higher  order  skills)  were  considered  critical  for  effective  performance  in  this 
environment:  Communication,  Situation  Awareness,  Adaptability/Flexibility,  and  Coordination. 
Each  of  the  four  dimensions  was  defined  and  then  decomposed  into  key  factors. 
“Communication,”  for  example,  was  defined  as  “the  exchange  of  information  between  two  or 
more  team  members”  and  was  decomposed  into  four  key  factors:  information  exchange — clear, 
concise,  and  accurate  exchange  of  information  between  elements  of  the  team;  information 
clarification — detection  of  inaccurate  or  incomplete  information  and  taking  corrective  action; 
information  cadence — timing,  rhythm,  and  flow  of  information;  and,  information  format — 
terminology  and  order  of  the  information.  The  definitions  and  key  factors  associated  with  each 
of  the  dimensions  are  listed  in  Table  2.  For  each  of  the  dimensions,  evaluators  were  to  provide  a 
rating  for  each  mission  phase  using  a  5-point  scale  with  the  anchors:  1 — Dangerous,  2 — Below 
Average,  3 — -Nominal,  4 — Above  Average,  and  5 — Extraordinary.  The  observers  were  also 
asked  to  provide  an  overall  mission  rating  for  each  of  the  dimensions  and  an  overall  mission 
grade  using  the  same  scale.  A  sample  TOM  gradesheet  used  in  RoadRunner  '98  is  presented  in 
Table  3  a. 

Table  3a.  Gradesheet  used  to  assess  TOM  dimensions  for  each  mission  phase 


Phase/Dimension 

Communication 

Situation 

Awareness 

Adaptability/ 

Flexibility 

Crew 

Coordination 

Notes 

Brief 

Prior  to  Push 

Ingress 

Combat  Air 

Patrol  Formation 

Intercept 

Engagement 

Ground  Target 
Attack 

Egress 

Debrief 

Overall  Mission 
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Table  3b.  Sample  gradesheet  used  to  assess  team  performance  using  TARGET 
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Leaker  calls  (positive  or  possible) 
Identification  of  Unknowns 


The  second  approach  is  referred  to  as  Targeted  Acceptable  Responses  to  Generated  Event 
or  Tasks  (TARGET).  The  TARGET  approach  looks  at  proficiency  with  respect  to  scenario 
events.  Events  can  be  associated  with  a  specific  mission  phase  as  a  course  of  normal  operations 
(e.g.,  ingress,  ground  attack,  egress)  or  associated  with  situations  that  are  incorporated  into  a 
scenario  to  present  anomalous,  nonroutine,  or  unplanned  situations  (e.g.,  loss  of 
communications,  inaccurate  intelligence).  The  TARGET  approach  uses  behaviorally  focused 
scales  for  recording  observations  of  team  behaviors  for  given  events  using  the  same  1  to  5  scale 
as  TOM.  Events  are  arranged  on  the  gradesheet  in  the  approximate  order  in  which  they  are 
expected  to  occur  in  a  scenario.  A  sample  TARGET  gradesheet  used  in  RoadRunner  '98  is 
presented  on  Table  3b. 

AWACS  data.  Prior  to  the  RoadRunner  exercise,  38  expert  AW  ACS  weapons  directors 
were  interviewed.  These  subject-matter  experts  identified  four  teamwork  functions  that  are 
essential  to  AWACS  team  performance:  (a)  mission  planning,  (b)  communication 
content/timing,  (c)  adherence  to  support  situation  awareness,  and  (d)  supporting  behavior. 

Mission  planning:  formulation  of  contracts.  Mission  planning  affects  AWACS  teamwork 
by  establishing  roles  and  responsibilities,  and  through  contingency  planning.  As  discussed  by 
Fahey,  Rowe,  Dunlap  and  DeBoom  (in  press)  and  MacMillan  et  al.  (1998),  AWACS  weapons 
directors  explicate  roles,  responsibilities,  and  strategies  to  manage  team  member 
interdependencies  through  establishment  of  agreements  that  are  referred  to  as  contracts.  These 
contracts  are  made  among  AWACS  team  members  (internal  contracts),  and  to  the  external  team 
(i.e.,  pilots  of  friendly  assets). 

Communication:  adherence  to  protocol.  AWACS  weapons  director  tasks  are 
predominately  verbal  information  exchange.  Communications  are  heavily  standardized  in  terms 
of  content  (jargon)  and  process.  This  aspect  of  communication  effectiveness  refers  to  the  degree 
to  which  individuals  follow  guidelines  for  communication  exchange.  In  addition  to  proper 
jargon  and  syntax,  communications  must  be  clear,  concise,  and  correct. 

Communication  to  support  situation  awareness.  Communications  may  be  clear,  concise, 
correct,  and  delivered  according  to  proper  protocol,  and  still  be  superfluous.  Maintenance  of 
situation  awareness  also  requires  that  pertinent  information  be  exchanged  to  the  right  person,  at 
the  right  time.  In  such  a  communication-rich  environment,  too  much  communication  can  impede 
performance,  when  unnecessary  infonnation  interferes  with  other,  higher  priority 
communication.  Indeed,  part  of  the  proper  timing  of  AWACS  communications  to  pilots  involves 
knowledge  of  when  to  keep  quiet. 

Supporting  behavior.  AWACS  team  members  support  each  other  primarily  through 
updates  and  reminders  of  salient  information.  At  the  same  time,  they  can  also  transfer  resources 
(responsibilities)  and  confer  on  decisions/actions. 

A  rating  instrument  was  tailored  to  capture  these  dimensions  across  three  phases  of 
performance:  (a)  premission  briefing,  (b)  mission  performance,  and  (c)  mission  debriefing. 
Ratings  were  based  on  a  4-point  rating  scale.  Standards  for  performance  ratings  in  RoadRunner 
'98  were  the  same  as  used  in  AWACS  training: 

0  =  No  Ability  or  Knowledge.  Task  failure 

1  =  Lacks  Proficiency  (Coordination,  communication,  cohesion).  Uncorrected  errors. 
Degraded  mission  outcome  or  endangered  friendly  forces. 
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2  =  Limited  Proficiency.  Recognizes  and  corrects  errors  with  team  recovery.  Mission 
Degraded. 

3  =  Proficient.  No  mission  impacting  errors.  Team  reacts  correctly  in  current  situations. 

4  =  Highly  Proficient.  Prevents  errors.  Team  anticipates  future  situations.  Plans  ahead. 
NA  =  Not  Applicable 

NO  =  Not  Observed 
NP  =  Not  Performed 


Procedures 


Systems  design,  scenario  development,  and  network  testing  occurred  over  several  months 
prior  to  the  exercise.  Given  that  RoadRunner  '98  incorporated  four  active  sites  (AFRL, 
TACCSF,  AFIWC,  and  AW  ACS),  four  types  of  virtual  simulators  (F-15s,  F-16s,  MiGs,  and 
AWACS)  and  several  sources  of  constructive  entities,  much  of  the  development  effort  was 
devoted  to  testing  the  many  interactions  among  entities.  For  any  given  pair  of  fighter  aircraft, 
interactions  include:  visual  appearance,  radar  detection  and  tracking,  operation  of  each  cockpit’s 
RWR,  missile  lock-on  and  flyout,  and  effects  of  countermeasures  on  missiles. 

The  RoadRunner  '98  exercise  began  on  Monday,  13  Jul  98,  with  in-briefings  for 
participants  at  each  location,  administration  of  questionnaires,  and  a  familiarization  flight.  The 
goal  of  the  familiarization  flight  was  to  provide  participants  with  experience  in  the  simulators 
and  using  the  debrief  systems. 

On  Tuesday  through  Thursday  teams  flew  two  missions  per  day;  a  representative  daily 
schedule  is  shown  on  Table  lb.  Each  mission  began  with  a  one-hour  brief  at  each  location. 
Participants  were  to  be  in  their  cockpits  by  the  beginning  of  the  mission  period.  Scenarios  were 
designed  to  last  between  35  and  45  minutes  with  extra  time  allotted  in  the  schedule  for  slips  and 
restarts.  After  completion  of  the  mission,  90  minutes  was  allocated  for  debrief.  After  debrief, 
participants  completed  a  feedback  form  that  included  items  on  systems  difficulties  and  perceived 
training  effectiveness. 

The  RoadRunner  '98  exercise  concluded  on  Friday,  18  Jul  98,  with  one  mission.  After 
debriefing  this  mission,  teams  completed  a  questionnaire  on  their  assessment  of  DMT  training 
effectiveness. 
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RESULTS 


All  RoadRunner  '98  missions  were  completed  as  scheduled  without  aborts  or  significant 
delays  although  brief  delays  of  up  to  ten  minutes  were  not  uncommon.  On  two  occasions,  one  of 
the  F-16  cockpits  at  AFRL/HEA  failed  and  was  inserted  into  the  mission  after  rebooting  the 
simulator.  On  two  other  occasions,  one  cockpit  had  a  problem  that  could  not  be  corrected  and  the 
remaining  F-16s  continued  as  a  three-ship  formation.  Communication  bandwidth  among 
RoadRunner  '98  sites  was  not  a  limiting  factor.  For  the  DCA  mission,  which  had  the  highest 
number  of  entities  emitting  data  packets,  mean  network  traffic  was  315  kilobits  per  second  with  a 
peak  of  803  kilobits  per  second  which  is  slightly  over  one-half  the  capacity  of  a  T-l  line. 
Overall,  the  RoadRunner  '98  exercise  was  successfully  completed  as  planned.  The  following 
results  relate  to  mission  performance  within  the  exercise,  rated  training  effectiveness  of  DMT, 
and  areas  for  improvement  in  DMT  systems. 

Mission  Performance  and  Instructor  Evaluations 


The  first  and  last  missions  in  RoadRunner  '98  were  SAT  scenarios  (see  Appendix  A,  SAT 
1  and  SAT  2).  Two  teams  flew  SAT  1  on  Tuesday  and  SAT  2  on  Friday  while  the  other  team 
flew  the  scenarios  in  reverse  order.  Data  were  collected  for  the  virtual  F-16s  on  bomb  score 
(miss  distance),  red  air  (MiG)  kills,  and  losses  to  air  and  surface  threats.  Mean  bomb  score  for 
the  first  mission  was  425  feet  and  296  feet  for  the  last  mission.  Since  the  test  missions  were 
flown  shields-up,  virtual  F-16s  could  be  killed  more  than  once.  On  the  first  mission,  the  three 
teams  lost  a  total  of  15  aircraft  to  surface  threats,  13  to  airborne  threats,  and  killed  4  MiGs.  On 
the  last  mission,  virtual  F-16  teams  lost  9  aircraft  to  surface  threats,  6  to  airborne,  and  killed  no 
MiGs.  (Pilot  reactions  to  constructive  forces  are  also  described  in  Cicero,  1998.) 


□  SAT  1 
■  SAT  2 


TOM  Dimension 

Figure  9.  Mean  mission  performance  ratings 


A  rating  of  overall  mission  performance  on  a  scale  of  1  to  5  was  part  of  the  TOM  rating 
fonns  that  were  completed  by  each  team’s  instructor  pilot.  Mean  ratings  for  the  first  and  last 
SAT  missions  are  depicted  in  Figure  9.  The  mean  rating  assigned  to  overall  mission 
performance  for  the  three  teams  increased  from  1.7  to  3.2  between  the  first  and  last  missions.  In 
addition  to  a  rating  of  overall  mission  performance,  the  squadron  instructor  pilots  rated  each 
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team  on  Communication,  Situation  Awareness,  Adaptability/Flexibility,  and  Crew  Coordination 
(see  Figure  9).  Ratings  for  each  of  these  variables  for  all  teams  increased  from  approximately  2 
(below  average)  to  3  (nominal  performance)  except  for  Adaptability/Flexibility  which  was 
constant  at  3  for  two  teams  and  increased  from  2  to  3.5  for  only  one  team. 

Mean  ratings  for  overall  mission  performance  and  the  four  higher  order  skills 
demonstrated  by  team  A  for  the  two  SAT  missions  averaged  across  all  mission  phases  are 
depicted  in  Figure  10a.  The  rating  assigned  to  overall  mission  performance  for  team  A  increased 
from  1  (dangerous)  during  SAT  1  to  3  (nominal  performance)  during  SAT  2.  Mean  ratings  for 
each  of  the  higher  order  skills  also  increased  from  SAT  1  to  SAT  2.  In  an  effort  to  further 
examine  the  performance  of  team  A,  performance  by  mission  phase  was  examined.  Figure  10b 
depicts  the  average  ratings  across  the  four  higher  order  skills  by  mission  phase.  Performance 
during  brief,  ground  attack,  egress,  and  debrief  improved  from  SAT  1  to  SAT  2.  In  comparison, 
performance  prior  to  push  and  during  ingress  did  not  change.  Further  analyses  were  conducted 
to  determine  the  extent  to  which  performance  for  each  of  the  higher  order  skills  changed  by 
phase  from  SAT  1  to  SAT  2. 


I  SAT  1 
I  SAT  2 


Comm  SA  A/F  Coord  Overall 
Figure  10.  a)  Mission  perfonnance  ratings  for  team  A 


b)  Mission  performance  ratings  for  team  A  by  individual  mission  phases 
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Figure  1 1  shows  the  instructor  ratings  for  team  A  for  each  of  the  higher  order  skills  by 
phase  of  flight.  Comparing  performance  in  team  A  from  SAT  1  to  SAT  2:  (a)  ratings  for 
Communication  increase  for  brief,  ground  attack,  egress,  and  debrief  from  below  average  rating 
to  nominal,  whereas  ratings  for  prior  to  push  and  ingress  did  not  change  and  were  nominal;  (b) 
ratings  for  Situation  Awareness  increased  across  all  phases  except  for  the  ingress  from  below 
average  or  dangerous  to  nominal,  whereas  the  rating  for  ingress  decreased  from  nominal  to 
below  average;  (c)  ratings  for  Adaptability/Flexibility  increased  during  brief,  egress,  and  debrief 
from  below  average  to  nominal,  remained  at  a  nominal  level  for  prior  to  push  and  ingress,  and 
decreased  from  above  average  to  nominal  in  ground  attack',  and  (d)  ratings  for  Coordination 
increased  for  brief,  ground  attack,  egress,  and  debrief  from  below  average  to  nominal,  whereas 
ratings  for  prior  to  push  and  ingress  did  not  change  and  were  nominal. 

While  many  of  the  higher  order  skills  improved  from  SAT  1  to  SAT  2,  two  decreases  in 
ratings  were  observed  -  Situation  Awareness  during  ingress  and  Adaptability/Flexibility  during 
ground  attack.  The  previously  discussed  less  than  nominal  performance  observed  in  ingress  may 
have  been  the  result  of  below  average  performance  associated  with  Situation  Awareness  as 
compared  to  the  other  higher  order  skills  which  were  observed  to  be  nominal. 


Figure  11.  Skill  ratings  for  team  A  for  individual  mission  phases 


To  further  understand  the  manner  in  which  the  performance  of  Team  A  changed  as  a 
function  of  the  team’s  experience  in  DMT,  the  TARGETS  data  were  examined  at  the  task  level. 
Behaviors  were  identified  that  either  increased  or  decreased  by  one  or  more  points  from  SAT  1 
and  SAT  2.  Table  4  shows  that  20  behaviors  improved  from  SAT  1  to  SAT  2  whereas  only  one 
behavior  decreased  from  SAT  1  to  SAT  2. 
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Table  4.  Changes  in  TARGET  behaviors  for  Team  A 


Phase 

Performance  Increase  from 

SAT  1  to  SAT  2 

Performance  Decrease  from 

SAT  1  to  SAT  2 

Brief 

-  Review  contingencies  for  Rendezvous 

-  Mission  allowable  risk 

-  Commit  Criteria 

Prior  to 
Push 

-  AW  ACS  helps  develop  and  maintain 
picture  with  acceptable  timing  and 
cadence 

Ingress 

-  AWACS  provides  updated  picture 

-  Fighter  provides  updated  contact 
report  with  appropriate  terminology 
and  timing 

-  AWACS  helps  maintain  picture  with 
acceptable  timing  and  cadence 

-  Detection  of  G/A  Systems 

-  Avoidance  of  G/A  Systems 

-  Assess  implications  of  G/A  systems 

-  Push  Calls 

Ground 

Attack 

-  Detection  of  G/A  Systems 

-  Avoidance  of  G/A  Systems 

-  Assess  implications  of  G/A  systems 

Egress 

-  Detection  of  G/A  Systems 

-  Avoidance  of  G/A  Systems 

-  Assess  implications  of  G/A  systems 

Debrief 

-  Communication 

-  Situation  Awareness 

-  Adaptability/Flexibility 

-  Coordination 

Mean  ratings  for  overall  mission  performance  and  the  four  higher  order  skills 
demonstrated  by  team  B  for  the  two  SAT  missions  averaged  across  mission  phases  are  depicted 
in  Figure  12a.  The  rating  assigned  to  overall  mission  performance  for  team  B  during  SAT  1 
increased  from  2  (below  average)  to  3  (nominal  performance)  during  SAT  2.  Mean  ratings  for 
all  of  the  higher  order  skills  also  increased  from  SAT  1  to  SAT  2.  Figure  12b  depicts  the  average 
ratings  across  the  four  higher  order  skills  by  mission  phase.  Performance  ratings  for  all  phases 
improved  from  SAT  1  to  SAT  2  except  ground  attack  which  decreased.  In  SAT  1 ,  performance 
for  all  mission  phases  was  rated  at  or  below  average.  In  comparison,  performance  in  SAT  2  was 
rated  above  average  for  prior  to  push,  egress,  and  debrief,  nominal  for  briefing  and  ingress,  and 
slightly  below  average  for  ground  attack.  Additional  analyses  were  performed  separately  for 
each  of  the  higher  order  skills  by  phase  for  SAT  1  and  SAT  2.  Figure  13  shows  the  ratings 
provided  by  the  instructor  pilot  for  team  B  for  each  of  the  higher  order  skills  by  phase  of  flight. 
Comparing  performance  in  SAT  1  to  SAT  2:  (a)  ratings  for  Communication  increased  across  all 
phases  except  for  ground  attack  from  nominal  or  below  average  to  nominal  or  above  average,  in 
comparison  ratings  for  ground  attack  decreased  from  nominal  to  below  average;  (b)  ratings  for 
Situation  Awareness  increased  for  prior  to  push,  ingress,  egress,  and  debrief  from  nominal  or 
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below  average  to  nominal  or  above  average,  ratings  remained  nominal  for  brief,  and  ratings  for 
ground  attack  decreased  from  nominal  to  below  average;  (c)  ratings  for  Adaptability/Flexibility 
increased  for  brief,  prior  to  push,  ingress,  and  egress  from  nominal  or  below  average  to  nominal 
or  above  average,  ratings  for  debrief  remained  nominal,  and  ratings  for  ground  attack  decreased 
from  nominal  to  below  average;  and  (d)  ratings  of  Coordination  for  prior  to  push,  ingress,  and 
egress  increased  from  nominal  or  below  average  to  nominal  or  above  average,  and  ratings  for 
brief  and  ground  attack  remained  nominal.  While  many  of  the  higher  order  skills  improve  or 
remain  stable  from  SAT  1  to  SAT  2,  ratings  in  the  ground  attack  for  Communications,  Situation 
Awareness,  and  Adaptability  decreased  for  team  B.  The  previously  identified  less  than  nominal 
performance  observed  in  ground  attack  may  have  resulted  from  below  average  performance 
associated  with  these  higher  order  skills  as  compared  to  Coordination. 


Figure  12.  a)  Mission  performance  ratings  for  Team  B 
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b)  Mission  performance  ratings  for  Team  B  by  individual  mission  phases 


The  TARGETS  data  were  examined  to  identify  specific  task-related  behaviors 
demonstrated  in  both  SAT  1  and  SAT  2  that  increased  or  decreased  by  one  or  more  points. 
Table  5  shows  that  ratings  for  43  behaviors  improved  from  SAT  1  to  SAT  2  whereas  ratings  for 
only  three  behaviors  decreased  from  SAT  1  to  SAT  2.  All  other  behaviors  either  did  not  change 
or  were  not  demonstrated  in  both  sessions.  Almost  50%  of  the  behaviors  that  team  B  improved 
from  SAT  1  to  SAT  2  were  associated  with  the  brief  phase  of  the  mission. 
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Figure  13.  Skill  ratings  for  Team  B  for  individual  mission  phases 


Table  5.  Changes  in  TARGET  behaviors  for  Team  B 


Phase 

Performance  Increase  from 

SAT  1  to  SAT  2 

Performance  Decrease  from 
SAT  1  to  SAT  2 

Brief 

-  Review  of  tasking,  mission,  intelligence  brief 

-  Review  ROE 

-  Review  route,  entry  point,  target  area,  egress 

-  Review  boundaries,  FEBA 

-  Review  ordnance  loadout 

-  Review  rendezvous  procedures 

-  Review  contingencies  for  rendezvous 

-  Preferred  calls 

-  Cadence/discipline 

-  Time  line 
-TOT 

-  Speed/altitude 

-  Mission  allowable  risk 

-  Flow 

-  PID  engagement  criteria 

-  Blue  on  blue  deconfliction 

-  Commit  Criteria 

-  Review  of  roles  and  responsibilities  of  elements 

-  Review  of  procedures  for  handling  threats 

-  Egress  speed/altitude 

-  Egress  deconfliction  plan,  and 

-  Egress  Communication  plan 

-  Joker  and  Bugout  Calls 
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Prior  to 
Push 

-  Perform  fighter  to  AW  ACS  brief 

-  AWACS  to  fighter  brief 

-  Alpha  check 

-AWACS  provides  initial  and  updated  picture 

-  AWACS  helps  develop  and  maintain  picture  with 
acceptable  timing  and  cadence 

-  Discuss  implications  for 
ingress/egress  routes 

Ingress 

-  AWACS  helps  to  maintain  picture  with 
acceptable  timing  and  cadence 

-  Discuss  implications  for  ingress/egress  routes 

-  Push  calls 

-  Detection  of  G/A  Systems 

-  Avoidance  of  G/A  Systems 

-  Fighter  provides  descriptive  calls  with  acceptable 
timing  and  cadence 

-  Radar  search,  detection,  lock  on 

Attack 

-  Detection  of  G/A  Systems 

-  Avoidance  of  G/A  Systems 

-  Positive  Identification  of 

Target 

Egress 

-  Detection  of  G/A  Systems 

-  Avoidance  of  G/A  Systems 

-  Assess  implications  of  G/A  systems 

Debrief 

-  Communication 

-  Situation  Awareness 

-  Adaptability/Flexibility 

-  Coordination 

Mean  ratings  for  mission  performance  and  higher  order  skills  demonstrated  during  the 
two  SAT  missions  by  team  C  are  depicted  in  Figure  14a.  The  rating  assigned  to  overall  mission 
performance  for  team  C  during  SAT  1  increased  from  2  (below  average)  to  3  (nominal 
performance)  during  SAT  2.  Mean  ratings  for  all  four  of  the  higher  order  skills  also  increased 
from  SAT  1  to  SAT  2.  Ratings  of  team  C’s  performance  on  the  higher  order  skills  by  mission 
phase  are  shown  on  Figure  14b.  Performance  for  ingress,  ground  attack,  and  egress  improved 
from  SAT  1  to  SAT  2,  whereas  ratings  for  brief,  prior  to  push,  and  debrief  did  not  change. 
During  SAT  1,  performance  was  rated  nominal  for  brief,  prior  to  push,  and  debrief,  however 
ratings  were  below  average  for  ingress,  ground  attack,  and  egress.  In  comparison,  during  SAT 
2,  performance  was  nominal  for  all  phases. 
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Figure  14.  a)  Mission  performance  ratings  for  Team  C 
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b)  Mission  performance  ratings  for  Team  C  by  individual  mission  phases 


Figure  15  shows  instructor  ratings  for  team  C  for  each  of  the  higher  order  skills  by  phase 
of  flight.  Comparing  performance  in  SAT  1  to  SAT  2:  (a)  ratings  for  Communication  increase 
for  ingress,  ground  attack,  and  egress  from  dangerous  or  below  average  to  nominal,  whereas 
ratings  for  brief,  prior  to  push,  and  debrief  remained  nominal;  (b)  ratings  for  Situation 
Awareness  increased  for  ingress  and  ground  attack  from  dangerous  or  below  average  to  nominal 
while,  ratings  for  brief,  prior  to  push,  egress,  and  debrief  remained  nominal;  (c)  ratings  for 
Adaptability/Flexibility  increased  for  ingress  and  ground  attack  from  below  average  to  nominal, 
whereas  brief,  prior  to  push,  egress,  and  debrief  remained  nominal;  and  (d)  ratings  for 
Coordination  increased  for  ingress,  ground  attack,  and  egress  from  dangerous  or  below  average 
to  nominal,  whereas  ratings  for  briefing,  prior  push,  and  debrief  remained  nominal.  All  of  the 
ratings  for  the  higher  order  skills  increased  or  remained  stable  from  SAT  1  to  SAT  2  and  none  of 
the  ratings  for  team  C  decreased. 


Figure  15.  Skill  ratings  for  Team  C  for  individual  mission  phases 


Table  6  shows  that  1 5  TARGET  behaviors  were  improved  from  SAT  1  to  SAT  2  whereas 
three  behaviors  decreased  from  SAT  1  to  SAT  2. 
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Table  6.  Changes  in  TARGET  behaviors  for  Team  C 


Phase 

Performance  Increase  from 

SAT  1  to  SAT  2 

Performance  Decrease  from 

SAT  1  to  SAT  2 

Brief 

-  Review  Rules  of  Engagement 

Prior  to 
Push 

-  AW  ACS  provides  initial  and  updated 
picture 

Ingress 

-  AW  ACS  provides  update  picture 

-  Acknowledge  and  respond  to 

AWACS  as  appropriate 

-  AWACS  helps  maintain  picture  with 
acceptable  timing  and  cadence 

-  Fighter  provides  updated  contact 
report 

-  Detection 

-  Tactical  considerations  discussed 

-  Leaker  calls  (positive  or  possible) 

Attack 

-  Delivery  parameters 

-  Request  Re-Attack,  if  necessary 

-  Detection  of  G/A  Systems 

-  Avoidance  of  G/A  Systems 

-  Assess  implications  of  G/A  systems 

-  Positive  Identification  of  Target 

Egress 

Debrief 

-  Communication 

-  Situation  Awareness 

-  Coordination 

The  behavioral  changes  obtained  on  the  TARGETS  instrument  were  examined  to 
determine  the  number  of  teams  demonstrating  positive  or  negative  change  of  one  or  more  points 
for  a  given  behavior  in  each  phase.  Table  7  lists  the  behaviors  by  mission  phase  that  improved  in 
two  or  three  teams  from  SAT  1  to  SAT  2.  Seven  behaviors  improved  across  all  three  teams 
across  three  different  mission  phases.  In  comparison,  only  one  behavior  decreased  in  2  of  the  3 
teams  (i.e.,  positive  identification  of  target  during  ground  attack  phase)  and  none  of  the 
behaviors  were  observed  to  decrease  for  all  three  of  teams. 

Table  7.  Summary  of  changes  in  TARGET  behaviors 


Phase 

Improvements  in  3  of  3  teams 

Improvements  in  2  of  3  teams 

Brief 

Review  contingencies  for 
rendezvous 

Mission  allowable  risk 

Commit  Criteria 

Prior  to 

Push 

AWACS  provides  initial  and 
updated  picture 

AWACS  helps  develop  and 
maintain  picture  with  acceptable 
timing  and  cadence 
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Table  7.  Cont. 


Ingress 

AWACS  helps  maintain  picture 
with  acceptable  timing  and  cadence 
Detection  of  G/A  systems 

AWACS  provides  updated  picture 
Avoidance  of  G/A  systems 

Fighter  provides  updated  contact 
report 

Ground 

Attack 

Detection  of  G/A  systems 

Avoidance  of  G/A  systems 

Assess  implications  of  G/A  systems 

Egress 

Detection  of  G/A  systems 

Avoidance  of  G/A  systems 

Assess  implications  of  G/A  systems 

Debrief 

Communication 

Situation  Awareness 

Coordination 

Adaptability/Flexibility 

Pilot  Aircraft  and  DMT  Evaluations 


Pilots  rated  the  effectiveness  of  their  current  aircraft  training  program  for  a  number  of 
mission  tasks  and  elements  at  the  beginning  of  RoadRunner  '98  and  repeated  this  questionnaire 
for  DMT  training  at  the  end  of  the  week. 

F-15  results.  F-15  pilots  in  RoadRunner  '98  rated  their  current  aircraft  training  program 
as  most  effective  for  improving  skills  in  Beyond  Visual  Range  Employment,  Radar  Lookout,  and 
Radar  Sorting  and  Employment.  Aircraft  training  was  rated  as  least  effective  for  Tactical 
Electronic  Warfare  System  Employment  and  for  Reaction  to  AAA  (Figure  16a).  DMT  was  rated 
as  highly  effective  for  training:  Radar  Sorting  and  Employment,  Electronic  Identification, 
Intraflight  Communication,  and  Engagements  Against  Multiple  (  >  4  )  Bogeys.  DMT  was  rated 
as  ineffective  for  training  Gun  Employment,  Visual  Identification,  Basic  Fighter  Maneuvers,  and 
Operating  in  a  Communications  Jamming  Environment  (Figure  16b). 

The  difference  between  the  mean  ratings  for  aircraft  training  and  DMT  was  computed  for 
each  mission  task  or  element  and  plotted  on  Figure  17.  Negative  values  on  this  figure  indicate 
that  training  in  the  aircraft  was  rated  as  more  effective  than  DMT  for  a  given  task  or  element. 
Aircraft  training  was  rated  as  much  more  effective  than  DMT  for  most  of  the  rated  tasks 
including  Tactical  Formation,  Visual  Lookout,  Mutual  Support,  and  All-Aspect  Defense. 
Positive  values  indicate  that  ratings  of  training  effectiveness  for  DMT  were  higher  than  ratings 
assigned  to  aircraft  training.  Only  Engagements  Against  Multiple  Bogeys,  Intraflight 
Communications,  and  Reaction  to  AAA  were  assigned  higher  effectiveness  ratings  for  DMT 
than  for  aircraft  training. 

F-16  results.  F-16  pilots  rated  their  current  aircraft  training  programs  as  most  effective 
for  Tactical  Formation,  Tactics  and  Mission  Planing,  Debriefing,  Radar  Targeting  and  Sorting, 
and  Visual  Missile  Employment  (Figure  18a).  Aircraft  training  was  rated  as  least  effective  for 
Engagements  Against  Multiple  Bogeys,  Intercepts  using  Electronic  Countermeasures,  Reaction 
to  AAA,  and  Employment  of  Chaff  and  Flares.  DMT  was  rated  as  most  effective  for  Radar 
Targeting  and  Sorting,  Engagements  Against  Multiple  Bogeys,  and  Radar  Lookout  /  Mechanics 
and  least  effective  for  Visual  Lookout,  Visual  Identification,  Basic  Fighter  Maneuvers,  and 
Tactical  Formation  (Figure  18b). 
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Mission  element  or  task  2?  Mission  element  or  task 


BVR  employment 
Radar  sort  &  employ 
Radar  lookout 
Electronic  ID 
Missile  employ 
All  aspect  defense 
Mutual  support 
Tactical  formation 
Tactical  intercept 
Egress  tactics 
2-ship  tactics 
Intraflight  comm 
Employ  chaff /flare 
4-ship  tactics 
Separation 
Medium  alt.  tactics 
Multibogey,  4+ 
Tactics  /  mission  pi 
Work  w/  AWACS  /  GCI 
DACT 
Debriefing 
Visual  lookout 
Gun  employ 
Reaction  to  Als 
ACM/ACT 
Visual  ID 
BFM 

Force  proledion 
Low  alt  tactics 
Reaction  to  SAMs 
Visual  low  level 
Comm  jam 
Safe  passage 
Reaction  to  AAA 
TEWS  employment 
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Aircraft  training  value 


gure  16.  a) 


F-15  pilot  ratings  for  aircraft  training  effectiveness 
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Reaction  to  AAA 
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Debriefing 
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DMT  training  value 

b)  F-15  pilot  ratings  for  DMT  effectiveness 
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Figure  17.  Differences  between  F-15  pilot  ratings  for  aircraft  training  and  DMT  effectiveness 
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Figure  18.  a)  F-16  pilot  ratings  for  aircraft  training  effectiveness 


Comparison  of  the  ratings  assigned  for  each  task  or  element  shows  that  aircraft  training 
was  rated  as  more  effective  than  DMT  for  several  tasks  including  Basic  Surface  Attack,  Mutual 
Support,  Visual  Lookout,  and  Tactical  Formation  (Figure  19).  Ratings  of  training  effectiveness 
assigned  to  DMT  were  higher  that  ratings  for  aircraft  training  for  Engagements  Against  Multiple 
Bogeys,  Employment  of  Chaff  and  Flares,  Reaction  to  AAA,  Multiship  Tactics,  and  Reaction  to 
SAMs.  F-16  pilots  qualified  as  wingmen  and  element  leads  assigned  higher  ratings  for  DMT 
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than  aircraft  training  on  Work  with  AW  ACS,  Radar  Targeting  and  Sorting,  and  Radar 
Mechanics.  Instructor  Pilots  and  Mission  Commanders  assigned  higher  ratings  to  DMT  for  All¬ 
aspect  defense,  and  Multiship  Package  Employment. 
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Figure  18.  b)  F-16  pilot  ratings  for  DMT  effectiveness 
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Comparison  of  DMT  and  aircraft  training 

Figure  19.  Differences  between  F-16  pilot  ratings  for  aircraft  training  and  DMT  effectiveness 
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AW  ACS  Performance  and  DMT  Evaluations 


AW  ACS  team  performance  was  rated  by  an  AWACS  instructor  for  Mission  Briefing, 
Mission  Execution,  and  Debriefing.  Mean  performance  ratings  for  the  SAT  1  and  SAT  2 
missions  are  depicted  on  Figure  20a.  More  detailed  analyses  of  AWACS  performance  ratings 
are  presented  in  Appendix  C. 

In  addition  to  instructor  evaluations,  AWACS  weapons  directors  were  asked  to  rate  the 
effectiveness  of  DMT  overall  and  in  comparison  to  large-force  exercises  such  as  Red  Flag. 
Overall  training  effectiveness  was  assessed  on  a  scale  from  1  to  5  with  the  anchors: 

1  =  Completely  useless 

2  =  Learned  a  little 

3  =  Somewhat  beneficial 

4  =  Very  useful 

5  =  Excellent  training 

Mean  ratings  are  depicted  on  Figure  20b.  In  comparison  to  large-force  exercises,  50%  or 
more  of  the  weapons  directors  participating  in  RoadRunner  '98  rated  DMT  as  providing  equal  or 
better  training  than  flag  exercises  for  all  missions  except  CAS.  Only  30%  of  weapons  directors 
and  air  surveillance  technicians  rated  DMT  as  providing  equal  or  better  training  for  the  CAS 
mission.  For  the  Interdiction  and  OCA  missions,  70%  of  weapons  directors  rated  DMT  training 
as  equal  or  more  effective  then  flag  exercises. 


Figure  20.  a)  Mean  AWACS  mission  performance  ratings 
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b)  AWACS  training  effectiveness  ratings 
Participant  Critiques  and  Feedback 

Pilots  and  AWACS  controllers  were  asked  to  critique  their  experience  in  RoadRunner  '98 
and  to  suggest  how  DMT  systems  should  be  used  in  future  training  exercises.  The  feedback 
provided  by  participants  can  be  grouped  into  three  overall  categories:  best  features  and 
applications  of  DMT,  system  problems,  and  factors  that  reduced  training  effectiveness. 

Best  features  and  applications  of  DMT.  The  participants’  overall  evaluation  was  that  the 
RoadRunner  '98  exercise  provided  valuable  experience  performing  combat  missions  and  tasks 
that  are  infrequently  practiced  in  the  air.  The  most  frequently  cited  strength  of  the  training  was 
the  capability  to  employ  as  a  four-ship  formation  against  multiple,  maneuvering  targets.  Pilots 
cited  targeting  and  sorting,  missile  employment,  communication,  and  building  situation 
awareness  as  significant  training  benefits.  AWACS  crews  also  cited  the  communication  with 
pilots  as  a  strong  point.  Weapons  directors  were  emphatic  about  the  training  value  of  briefing 
and  debriefing  with  the  Mission  Commander  even  though  these  contacts  were  limited  to  fax  and 
telephone;  Air  Surveillance  Technicians  were  much  less  positive  regarding  debrief.  According 
to  the  Weapons  Directors,  the  major  benefit  of  debriefing  with  pilots  was  receiving  direct 
feedback  on  the  quality  and  timeliness  of  their  radio  communication.  F-16  pilots  cited  the  replay 
and  debrief  system  at  AFRL,  Mesa,  as  being  valuable  for  increasing  their  skills  at  building 
situation  awareness  (see  Figures  6  a  and  b).  Pilots  stated  that  the  ability  to  simultaneously 
review  the  plan-view  display  and  each  cockpit’s  radar  and  RWR  was  beneficial  in  diagnosing 
errors.  Replay  of  the  stores  management  system  was  much  less  valuable  and  replay  of  the  HUD 
was  largely  illegible.  F-16  pilots  also  commented  that  the  air-to-air  scenarios  that  were 
presented  after  the  main  training  mission  was  completed  (see  Appendix  B)  provided  valuable 
training.  After  the  first  day  of  RoadRunner  '98,  the  F-16  pilots  asked  if  the  short  air-to-air 
scenarios  could  be  recorded  for  replay  and  debrief  and  the  day  after  that  the  AWACS  teams 
asked  if  they  could  participate.  Both  pilots  and  AWACS  teams  stated  that  these  brief  scenarios 
provided  excellent  training  for  multiship,  multibogey  air  combat.  Participants  stated  that  radar 
mechanics,  communication  skill,  tactical  execution,  and  situation  awareness  benefited  from  these 
scenarios.  Participants  further  stated  that  opportunities  to  practice  these  skills  on  the  training 
range  were  infrequent  and  that  experience  in  DMT  should  significantly  enhance  their  skills. 
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Recommendations  for  best  applications  of  DMT  focused  on  tasks  that  are  well  supported 
by  DMT  system  capabilities  and  are  infrequently  trained  in  the  aircraft.  These  tasks  involve 
communication  and  coordination  among  multiple  players  and  are  not  dependent  on  out-the- 
window  visual  imagery.  Multiship,  multibogey,  beyond-visual-range  air  combat  was  cited  by  all 
participants  as  the  best  application  for  DMT.  Pilots  also  cited  threat  reaction  and  all-aspect 
missile  defense  as  potentially  high  payoff  applications  but  that  limitations  in  constructive  forces 
in  RoadRunner  '98  limited  training  effectiveness.  Air-to-surface  ordnance  delivery  tasks  that 
were  major  components  of  the  RoadRunner  '98  SAT  and  OCA  A-G  missions  were  cited  as 
having  low  payoff  because  DMT  visual  capabilities  limit  training  effectiveness  and  these  tasks 
are  frequently  practiced  in  the  aircraft.  The  ordnance  delivery  component  of  the  RoadRunner  '98 
CAS  mission  was  limited  by  the  visual  display  system,  however,  coordination  with  the  A- 10 
FAC  was  cited  as  highly  beneficial.  All  participants  recommended  that  DMT  exercises  be 
structured  as  two-  or  more-day  events  rather  than  a  single  simulator  session.  The  multiple 
sessions  provided  opportunities  to  apply  and  reinforce  lessons  learned;  however,  the  relationship 
among  simulator  sessions  should  be  better  coordinated  than  in  RoadRunner  '98. 

System  problems.  The  large  majority  of  comments  identified  systems  or  simulations  that 
did  not  operate  correctly  or  like  the  real  world.  Many  of  these  items  could  be  corrected  during 
RoadRunner  '98  but  others  could  not.  Items  that  were  corrected  within  an  engagement  included 
re-boresighting  a  head-tracker  in  an  F-16/M2DART  or  restarting  an  image  generator  that  failed. 
Items  that  could  not  be  corrected  but  caused  minor  difficulties  included  switches  that  traveled  in 
the  incorrect  orientation  and  a  nonfunctional  Data  Entry  Display  in  one  F-16  cockpit.  Other 
problems  mentioned  in  critiques  resulted  in  significant  mission  impacts.  These  include  the 
communication  problems  between  the  cockpits  and  AWACS,  communications  failures  within 
the  AWACS  simulator,  and  the  F-16  replay  system  did  not  rewind  or  fast  forward  reliably. 
Fidelity  of  the  F-16  RWR  was  cited  as  a  serious  limitation  on  training.  The  simulators’  RWRs 
did  not  act  like  the  pilots  expected  and  as  a  result  they  were  unable  to  use  information  from  the 
RWR  to  effectively  defend  themselves. 

Training  effectiveness.  Several  issues  were  repeatedly  mentioned  as  limitations  to 
effective  training  using  RoadRunner  '98  systems.  Two  items  generated  most  of  comments 
regarding  training  effectiveness:  visual  display  systems  and  the  behavior  of  constructive  forces. 

The  F-15  WTTs  used  at  TACCSF  provided  only  forward  view,  out-the  window  visual 
display  that  was  combined  with  the  HUD.  Pilots  found  this  system  only  useful  for  providing  a 
horizon  and  an  earth-sky  frame  of  reference.  Training  effectiveness  was  limited  to  tasks  that 
could  be  conducted  inside  the  cockpit  such  as  beyond  visual  range  air  engagement.  The  general 
consensus  of  the  F-15  pilots  was  that  the  systems  they  experienced  in  RoadRunner  '98 
represented  a  step  in  the  right  direction  but  were  not  capable  of  providing  useful  training.  The  F- 
16  cockpits  at  AFRL/HEA  provided  a  wide  field-of-view,  full  field-of-regard  display  with  high 
brightness  and  contrast.  Pilots  described  the  system  as  providing  a  high  level  of  immersion  into 
the  battlespace  particularly  when  integrated  with  the  SGI  Reality  Monster  image  generator. 
Those  who  were  familiar  with  the  Red  Flag  training  range  found  that  they  could  easily  navigate 
using  the  imagery  provided  by  the  M2D ART/SGI  system.  Comments  regarding  the  adequacy  of 
this  display  for  air-to-surface  missions  were  decidedly  mixed.  Pilots  stated  that  the  wide  field-of 
view  helped  support  situation  awareness  and  allowed  them  to  employ  many  ordnance  deliveries 
such  as  a  pop-up  attack. 
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Target  resolution  was  not  cited  as  a  problem  when  the  Designated  Mean  Point  of  Impact 
(DMPI)  was  surrounded  by  larger,  easily  seen,  visual  features  such  as  runways  and  storage  tanks. 
For  the  CAS  mission,  however,  the  targets  were  surface  vehicles  such  as  T-72  tanks;  to  increase 
visibility,  the  tanks  were  moving  across  the  terrain  and  generating  dust  trails.  Many  pilots  noted 
that  even  with  the  dust  trails,  the  tanks  could  not  be  detected  from  altitude  and  that  they  often 
began  an  attack  against  the  coordinates  provided  by  the  A- 10  FAC  without  a  target  in  view. 
There  was  a  difference  of  opinion  on  this  point  between  more  and  less  experienced  F-16  pilots. 
The  less  experienced  pilots  cited  the  inability  to  clearly  detect  the  target  from  altitude  as  limiting 
the  training  value  of  the  M2DART  system.  More  experienced  pilots,  however,  noted  that  they 
often  are  unable  to  see  a  tank-sized  target  from  altitude  when  the  weather  is  less  than  ideal. 
Under  these  conditions,  pilots  must  use  the  cues  provided  by  the  FAC  describing  the  target’s 
location  with  respect  to  larger,  more  easily  seen  visual  features.  These  pilots  cited  the  M2DART 
as  providing  sufficient  resolution  for  training  CAS  missions  as  the  pilots  were  required  to  use  all 
the  visual  cues  in  the  environment  to  locate  their  target.  The  A- 10  pilots,  who  flew  a  cockpit 
equipped  with  an  earlier  generation  DART  visual  system,  evaluated  the  simulation  as  fully 
satisfactory  and  supported  performance  of  their  mission  as  a  FAC. 

The  most  significant  problems  for  the  M2DART  were  in  the  air-to-air  environment. 
Pilots  consistently  complained  that  they  could  not  determine  the  aspect  orientation  of  other 
aircraft  at  realistic  distances.  This  limitation  prevented  them  from  maintaining  tactical 
formation,  conducting  within-visual-range  engagements,  and  maintaining  visual  mutual  support. 
The  difficulty  was  that  the  red-blue-white  light  system  described  earlier  supported  pilots’  ability 
to  detect  other  aircraft  but  prevented  them  from  discriminating  which  way  the  other  aircraft’s 
nose  was  pointing,  whether  the  aircraft  was  flying  wings-level  or  turning,  and  whether  it  was 
approaching  or  moving  away.  This  limitation  was  a  source  of  frustration  that  did  not  decrease 
over  the  five  days  of  RoadRunner  '98. 

The  second  major  concern  expressed  in  pilot  critiques  was  the  behavior  of  the 
constructive  forces.  Both  airborne  and  surface  threats  were  cited  as  unrealistic  with  respect  to 
modeling  the  actual  systems  and  the  behavior  of  red  force  warfighters.  Comments  regarding 
physical  modeling  included  the  long  range  and  uncanny  accuracy  of  red  missiles,  difficulty  in 
defeating  red  missiles,  the  speed  and  turning  capabilities  of  red  aircraft,  their  apparently 
unlimited  fuel  reserves,  and  the  ability  of  surface  threat  systems  to  detect  and  track  blue  aircraft 
without  stimulating  their  warning  systems.  Comments  regarding  the  behavior  of  red  forces 
focused  on  their  very  high  level  of  situation  awareness,  speed  with  which  red  pilots  were  able  to 
sort  a  formation  and  launch  missiles,  and  their  ability  to  retarget  a  blue  formation  after  losing 
radar  lock.  These  factors  were  frequently  cited  as  a  major  limitation  on  training  effectiveness  as 
pilots  were  unable  to  employ  defensive  tactics  against  threats  that  had,  “360°  radar,  mach-10 
missiles,  and  perfect  situation  awareness.” 

In  addition  to  visual  displays  and  constructive  forces,  other  systems  were  cited  as  limiting 
training  effectiveness.  One  comment  made  by  both  pilots  and  AW ACS  teams  was  that  the 
simulated  radars  worked  much  better  than  actual  systems.  Pilots  and  weapons  directors  could 
detect  targets  at  long  range  and  track  them  continuously  without  losing  track  due  to  ground 
clutter. 

F-16  pilots  frequently  mentioned  that  the  fire  control  radar’s  Track  While  Scan  (TWS) 
mode  operated  much  better  than  in  the  aircraft  and  could  encourage  over-reliance  on  this  mode. 
Participants  stated  that  having  simulated  systems  that  work  too  well  reduces  training 
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effectiveness  since  it  eliminates  much  of  the  fog  and  friction  of  war.  Conversely,  while 
communication  breakdowns  between  pilots  and  AWACS  were  cited  as  system  problems,  some 
pilots  noted  that  these  failures  were  representative  of  real-world  communications  thus 
contributing  a  desirable  quantity  of  fog  and  friction. 

Another  issue  that  reduced  training  effectiveness  was  the  amount  of  time  available  for 
mission  briefing  and  system  setup  after  an  exercise  began  particularly  for  AWACS  teams.  While 
F-15  pilots  did  not  cite  the  assigned  time  line  as  a  problem,  F-16  pilots  often  wanted  more  time 
between  unfreeze  and  push.  AWACS  crews  consistently  stated  that  they  needed  more 
information  from  the  pilots  during  briefing  and  more  time  to  set  up  their  radar  scopes  after  the 
mission  started.  AWACS  crews  also  stated  that  the  procedures  for  declaring  a  target  as  friendly 
or  hostile  were  unrealistic  and  that  mission  scenarios  were  designed  for  training  pilots  with 
inadequate  consideration  of  AWACS  crews.  Air  surveillance  technicians  were  especially  critical 
on  this  point  as  they  had  only  limited  roles  in  the  RoadRunner  '98  scenarios. 

A  third  issue  dealt  with  the  schedule  (see  Table  1).  Flying  two  different  missions  per  day 
that  were  independent  from  each  other  was  cited  as  not  providing  focused  training.  Participants 
suggested  reducing  the  variety  of  missions  or  generating  an  integrated,  four-day  war  as  ways  to 
increase  training  effectiveness. 

DISCUSSION 

The  specific  objectives  for  RoadRunner  '98  were  to:  (a)  demonstrate  DMT  capabilities 
necessary  for  combat  mission  training,  and  (b)  validate  a  research  approach  for  DMT  training 
effectiveness  studies.  Data  were  collected  during  RoadRunner  '98  to  address  these  objectives  in 
two  ways:  mission  accomplishment  and  team  performance.  Analyses  of  mission 
accomplishment  include  whether  missions  could  be  successfully  accomplished  in  the  DMT 
environment,  what  systems  or  capabilities  enhanced  or  degraded  from  mission  accomplishment, 
and  participant  evaluations  of  the  training  value  for  these  missions.  Analyses  of  mission 
performance  include  changes  in  measures  of  mission  success  and  instructor  or  supervisor 
evaluations  of  team  perfonnance. 

Missions  and  Mission  Tasks 


RoadRunner  '98  was  designed  to  assess  how  well  prototype  systems  could  support  real¬ 
time  simulator-based  training  for  a  variety  of  composite  force  missions  over  a  secure,  wide-area 
network.  In  all  of  the  RoadRunner  '98  missions  other  than  CAS  and  DC  A,  F-15s  provided  a  pre¬ 
strike  sweep  while  the  F-16s  delivered  ordnance  against  a  defended  surface  target  and  AWACS 
provided  radar  surveillance  and  battle  management.  Other  blue  force  elements  such  as  SEAD 
aircraft  were  provided  constructively.  In  the  CAS  mission,  the  F-16s  and  the  A- 10  attacked  a 
fonnation  of  red  force  tanks  defended  by  AAA  while  the  F-15s  were  engaged  in  an  independent 
DCA  scenario  supported  by  AWACS.  In  the  DCA  scenario,  the  F-15s  and  F-16s  each  defended 
a  lane  of  airspace  from  red  force  airborne  threats  while  supported  by  AWACS.  In  all  cases,  the 
missions  were  completed  on  schedule  and  as  planned.  There  were  some  delays  and  on  occasion  a 
virtual  player  fell  off-line  and  the  mission  proceeded  without  them.  Overall,  RoadRunner  '98 
demonstrated  that  these  composite  force  missions  can  be  conducted  using  current  DMT  systems. 
More  detailed  examination  of  the  data  provides  insight  regarding  which  systems  enhanced  and 
degraded  mission  accomplishment  and  whether  the  exercise  provided  useful  training. 
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DMT  systems  and  mission  accomplishment.  Four  categories  of  DMT  systems  affected 
mission  accomplishment:  networking  and  communications,  weapons  systems  simulation,  visual 
displays,  and  constructive  forces. 

Networking  and  communications  using  DIS  protocols  and  dedicated,  high-bandwidth 
links  fully  supported  RoadRunner  '98  missions.  At  no  time  did  communications  bandwidth  or 
reliability  degrade  DMT  training  effectiveness  for  all  entities  other  than  AW  ACS.  Due  to  non¬ 
availability  of  high-bandwidth  communications,  a  work-around  solution  was  created  especially 
for  the  RoadRunner  '98  exercise  using  standard  telephone  lines.  This  system  suffered  from 
reliability  problems  although  these  decreased  over  the  course  of  the  exercise.  Training 
effectiveness  evaluations  conducted  subsequent  to  RoadRunner  '98  have  demonstrated  that 
AW  ACS  operations  can  be  fully  supported  by  standard  DIS  communications  protocols  and  high- 
bandwidth  links  (see  Crane,  1999).  It  must  be  noted,  however,  that  the  networking  system  was 
used  to  support  only  the  real-time  execution  of  the  mission.  Mission  planning,  briefing,  replay, 
and  debrief  were  conducted  using  an  independent  telephone  system. 

Weapons  systems  and  aircraft  systems  include  simulation  of  the  airframe  and  propulsion 
systems  together  with  avionics  and  weapons.  Simulation  of  systems  that  affected  only  individual 
aircraft  functions  such  as  handling  characteristics  were  rarely  cited  as  problems.  Cockpit 
systems  that  involved  interactions  among  different  entities  were  frequently  cited  as  degrading 
mission  accomplishment.  These  include  simulation  of  air-to-air  radar  that  at  times  performed 
both  much  better  than  and  much  poorer  than  the  actual  radar,  RWR  performance  that  did  not 
meet  expectations,  weapons  that  were  at  times  both  more  and  less  effective  than  specified  by 
doctrine,  and  ineffective  countermeasures.  The  critique  that  air-to-air  radar  performance  was 
superior  to  the  actual  aircraft’s  radar  reflects  problems  in  both  the  virtual  environment  and  on  the 
live,  training  range.  In  the  RoadRunner  '98  simulation,  effects  of  ground  clutter  were  not 
modeled  resulting  in  unrealistic  radar  performance  at  low  altitude.  Although  contrary  to  pilot 
expectations,  detection  ranges  for  threat  aircraft  at  medium  altitudes  and  higher  were  valid  for 
the  threat  aircraft  modeled  in  the  scenarios,  i.e.,  MiG-29s  and  Su-27s.  These  aircraft  have 
considerably  larger  radar  cross  sections  and  therefore  longer  detection  ranges  than  the  F-16s  that 
typically  play  the  role  of  threat  aircraft  on  home  station  training  ranges.  For  these  threats,  pilot 
expectations  regarding  air-to-air  radar  performance  were  influenced  by  constraints  on  range 
training.  Another  source  of  difficulty  was  the  myriad  of  systems  on-board  each  entity  that  could 
affect  another  entity.  One  aircraft’s  radar,  for  example,  will  detect  and  track  another  aircraft  and 
at  the  same  time  stimulate  the  target  aircraft’s  RWR.  Further,  if  fired  upon  by  a  radar-guided 
missile,  the  pilot  of  a  target  aircraft  can  defeat  that  missile  using  countenneasures  such  as  chaff. 
Alternatively,  the  pilot  can  attempt  to  defeat  the  attack  by  breaking  the  missile’s  radar  lock,  fly 
behind  a  mountain,  or  defeat  the  missile  kinematically  by  aggressive  maneuvers  at  end-game. 
During  RoadRunner  '98,  fighter  aircraft  were  generated  from  several  sources  both  virtual  and 
constructive.  Many  sources  were  legacy  systems  that  were  designed  for  other  purposes  and 
modified  for  participation  in  multiplayer,  real-time  simulation  exercises.  Some  of  these 
simulations  were,  for  example,  sensitive  to  chaff  while  others  were  not  and  some  radar  models 
could  be  occluded  by  terrain  while  others  could  “see  through  mountains.”  While  these 
simulations  were  fully  adequate  for  their  original  purposes,  they  lack  capabilities  required  for 
incorporation  into  DMT  for  aircrew  training.  The  lack  of  consistency  among  the  interactions 
between  players  seriously  degraded  mission  accomplishment  during  RoadRunner  '98  as 
participants  were  unable  to  predict  whether  a  given  action  would  or  would  not  successfully 
achieve  an  intended  goal. 
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Visual  display  systems  were  frequently  cited  as  both  enhancing  and  degrading  mission 
accomplishment.  The  F-15  WTTs  were  equipped  with  only  a  forward  view  display.  These 
cockpits  supported  beyond- visual-range  weapons  employment  but  little  else.  F-15  pilots  cited 
only  in-cockpit  tasks  such  as  radar  sorting  and  employment  and  non-visual  tasks  such  as 
intraflight  communications  as  being  well  trained  in  DMT. 

The  F-16  MTTs  were  equipped  with  a  full  field-of-view  display.  F-16  pilot  evaluations 
indicate  that  these  systems  supported  more  training  tasks  than  the  F-15  WTTs  such  as  reaction  to 
SAMs  and  surface  attack  tactics  missions.  Like  the  F-15  pilots,  however,  the  F-16  pilots  rated 
in-cockpit  and  non-visual  tasks  as  being  most  effectively  trained  using  DMT.  The  most 
significant  limitation  of  the  F-16  visual  display  systems  was  the  pilots’  inability  to  determine  the 
aspect  of  another  aircraft  at  real-world  ranges.  Pilots  had  difficulty  engaging  red  aircraft  and 
supporting  blue  aircraft.  F-16  pilots  rated  tactical  formation  as  being  the  task  most  effectively 
trained  in  the  aircraft  and  the  least  effectively  trained  in  DMT.  The  white-red-blue  colored  light 
system  used  during  RoadRunner  '98  was  useful  but  not  an  effective  substitute  for  real-world 
target  resolution.  The  capability  of  the  M2DART  to  support  air-to-surface  training  varied  with 
the  target  and  the  pilots’  expectations.  For  some  missions,  targets  were  selected  to  provide  large, 
easily  seen  visual  cues  such  as  a  runway  intersection  on  an  airfield.  The  pilots  did  not  find  using 
the  visual  display  systems  for  these  missions  to  be  problematic.  For  smaller  targets  such  as  tanks 
used  in  the  CAS  mission,  feedback  was  mixed  depending  on  pilot  experience  and  expectation. 
Overall,  factors  other  than  target  visibility  were  responsible  for  lack  of  mission  accomplishment 
on  air-to-surface  missions.  These  factors  include  inability  to  effectively  use  terrain  masking  and 
inability  to  maintain  tactical  formation  or  mutual  support  with  other  F-16s. 

The  M2D ART’S  full  field-of-regard  visual  display  system  was  also  cited  as  providing  the 
most  immersive  and  engaging  simulation  the  F-16  pilots  had  ever  experienced.  The  wide  field- 
of-view  enhanced  the  pilots’  ability  to  maintain  aircraft  control  by  using  their  peripheral  vision  to 
keep  track  of  the  horizon,  and  the  special  effects  such  as  explosions  enhanced  situation 
awareness.  Pilots  who  flew  the  M2DARTs  that  were  equipped  with  Silicon  Graphics  photo- 
based  Reality  Monster  image  generators  repeatedly  mentioned  the  realism  of  the  display  as 
enhancing  situation  awareness  for  the  environment.  Pilots  who  were  familiar  with  the  Red  Flag 
training  range  could  easily  navigate  using  the  photo-based  imagery  particularly  at  medium 
altitude. 

Constructive  forces.  Like  the  visual  display  systems,  the  behavior  of  constructive  forces 
both  enhanced  and  degraded  from  training  effectiveness.  F-15  and  F-16  pilots  were  near 
unanimous  is  rating  air  combat  in  a  multi-bogey  environment  as  the  task  most  effectively  trained 
in  DMT.  The  capability  of  DMT  to  provide  multiple,  maneuvering  MiG-23,  -29,  and  Su-27 
aircraft  was  the  most  highly  rated  training  feature  of  the  systems  in  RoadRunner  '98.  The 
behavior  of  these  target  aircraft  and  their  weapons  was  also  among  the  most  severely  criticized 
aspect  of  RoadRunner  '98.  The  main  thrust  of  the  critiques  was  that  the  red  force  aircraft  did  not 
die  when  they  should  have  and  had  unrealistic  combat  capabilities  including  radar  coverage, 
situation  awareness,  ability  to  attain  and  maintain  radar  lock,  turning  rate  and  radius,  and 
sustained  airspeed.  Many  of  these  criticisms  were  traced  in  post-exercise  reviews  to  incomplete 
threat  models.  For  example,  the  MiG-29s  could  maintain  very  high  airspeed  for  the  duration  of 
an  exercise.  Post-exercise  analysis  revealed  that  the  MiG’s  fuel  did  not  decrement  with  time  and 
airspeed  so  that  the  threat  aircraft,  in  effect,  flew  using  full  afterburner  all  the  time.  A  second 
example  was  the  unrealistically  high  probability  of  a  kill  (Pk)  for  the  constructive  SAMs. 
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Analyses  revealed  that  the  missiles  did  not  have  unrealistic  ranges  or  sensor  capabilities  but  that 
environmental  effects  which  would  reduce  the  missiles’  Pk  in  the  real  world  such  as  the  effects  of 
terrain  clutter  were  not  modeled.  The  effect  of  these  incomplete  threat  models  was  to  reduce  the 
virtual  players’  ability  to  defend  themselves.  Overall,  the  incorporation  of  constructive  forces 
was  among  the  most  positive  aspects  of  DMT  in  RoadRunner  '98.  Problems  with  the 
constructive-force  models  were  among  the  most  significant  limitations  in  the  exercise.  Overall, 
the  results  of  RoadRunner  '98  demonstrate  that  with  some  areas  for  improvement,  prototype 
DMT  systems  are  capable  of  supporting  a  variety  of  composite  force  missions. 

Training  effectiveness  and  utility.  A  second  area  of  concern  for  DMT  is  training 
effectiveness  and  utility.  The  focus  of  this  concern  is  not  whether  DMT  can  support  training  for 
a  given  mission  or  task  but  whether  there  are  sufficient  benefits  to  justify  using  DMT  for  training 
that  mission  or  task.  Three  sources  of  data  from  RoadRunner  '98  reflect  on  this  issue:  the 
availability  and  sufficiency  of  aircraft  training  for  a  given  mission  or  task,  the  effectiveness  of 
DMT  for  providing  training,  and  the  design  of  training  events  and  instruction  support  systems. 

F-16  pilots  noted  several  tasks  or  mission  elements  that  are  not  well  trained  in  the 
aircraft.  These  include  tasks  that  cannot  be  simulated  on  a  training  range  for  reasons  of  safety 
(e.g.,  reaction  to  AAA),  security  (e.g.,  employment  of  ECM/ECCM),  and  lack  of  resources  (e.g., 
multibogey  engagements).  Other  tasks  such  as  tactical  fonnation  and  visual  missile  employment 
were  rated  as  very  well  trained  in  the  aircraft.  Similarly,  F-16  pilot  ratings  of  training 
effectiveness  in  DMT  show  that  some  tasks  are  well  suited  for  training  in  DMT  such  as  radar 
targeting  and  sorting  and  that  other  tasks  such  as  tactical  formation  are  not.  Combining  pilot 
ratings  of  effectiveness  for  aircraft  training  and  DMT  reveals  some  mission  tasks  that  are  well 
supported  by  DMT  and  infrequently  trained  in  the  aircraft.  Multibogey  (4+)  engagement  was  the 
DMT  task  highest  rated  by  both  F-15  and  F-16  pilots.  F-16  pilots  also  included  threat  reaction, 
employment  of  chaff  and  flares,  multiship  package  employment,  and  work  with  AW  ACS  as 
being  both  well  supported  by  DMT  and  only  intermittently  trained  in  the  aircraft  due  to  safety, 
security,  and  resource  constraints.  These  data  indicate  that  DMT  training  utility  for  pilots  is  a 
function  of  both  the  quality  of  the  training  provided  in  DMT  and  the  availability  of  aircraft 
training.  Similar  results  can  be  seen  for  AWACS  teams.  Compared  to  Red  Flag,  Weapons 
Directors  and  Air  Surveillance  Technicians  rated  DMT  as  providing  effective  training  for 
missions  other  than  CAS.  In  addition,  Weapons  Directors  rated  DMT  as  more  effective  than  Red 
Flag  for  the  OCA  and  Interdiction  missions  in  which  blue  forces  fly  deeply  into  enemy 
controlled  airspace.  For  carefully  selected  tasks  and  missions,  therefore,  DMT  can  effectively 
complement  aircraft  training. 

A  second  factor  that  affects  training  effectiveness  and  utility  is  the  design  of  training 
events  and  instructional  support  systems.  In  order  to  assess  the  capabilities  of  DMT  to  support 
training  for  a  variety  of  missions,  RoadRunner  '98  incorporated  six  different  missions  within  four 
days.  One  F-16  pilot  referred  to  this  schedule  as  a  “smorgasbord  of  missions”  that  detracted 
from  training  effectiveness.  In  follow-up  interviews,  AWACS  teams  stated  that  incorporating 
missions  into  an  evolving  war  scenario  would  increase  the  training  value  of  a  DMT  exercise. 
Alternatively,  F-16  pilots  recommended  repeating  missions  morning  and  afternoon.  This 
schedule  would  give  them  the  opportunity  to  immediately  apply  the  lessons  learned  from  the 
morning  training  session.  This  compares  with  aircraft  training  in  which  a  pilot  may  learn  a 
valuable  lesson  but  not  experience  a  similar  mission  for  weeks  or  months. 
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RoadRunner  '98  participants  also  cited  mission  briefing,  replay,  and  debriefing  as 
significant  determinants  of  training  effectiveness.  AW  ACS  Weapons  Directors  stated  that  the 
opportunity  to  brief  and  debrief  with  the  pilots  immediately  before  and  after  missions  greatly 
enhanced  training  effectiveness.  This  compares  with  an  AWACS  aircraft  training  mission  in 
which  only  the  Senior  Controller  will  debrief  with  the  Mission  Commander  on  the  telephone 
several  hours  after  the  mission  is  over  and  the  AWACS  has  landed. 

Replay  and  debrief  in  RoadRunner  '98  consisted  of  independent  replays  of  a  mission 
followed  by  discussions  over  speaker  telephones  between  F-15,  F-16,  and  AWACS  teams. 
Although  these  discussions  became  heated  and  confrontational  at  times,  the  controllers  agreed 
that  they  learned  a  great  deal  about  what  information  pilots  need  and  when  they  need  it.  The  F- 
16  pilots  also  cited  quality  of  the  replay  system  as  enhancing  training  effectiveness.  The  replay 
system  at  AFRL/HEA  allowed  the  flight  leader  to  review  each  of  the  F-16s’  radars.  This 
capability  allowed  the  flight  leader  to  diagnose  and  remediate  errors  in  radar  mechanics  or 
failures  to  follow  the  briefed  search,  targeting,  and  sort  plans.  Replay  of  radio  communications 
for  all  players  allowed  participants,  particularly  the  flight  leaders  and  the  AWACS  controller,  to 
evaluate  and  remediate  their  calls.  Initially,  participants  focused  on  making  sure  that  their  calls 
were  correct,  complete,  concise,  and  in  accordance  with  Air  Force  standards.  As  the  week 
progressed,  the  emphasis  changed  to  insuring  that  communications  enhanced  the  flight’s  overall 
situation  awareness.  RoadRunner  '98  participants  agreed  that  telephone  communication  for 
debrief  was  insufficient  and  recommended  that  systems  be  provided  in  DMT  exercises  that 
would  allow  warfighters  at  multiple  locations  to  participate  more  extensively  in  mission  replay 
and  debrief.  Video  teleconference  capabilities  and  interactive  whiteboards  were  frequently 
suggested  options. 

Team  Performance 


Team  perfonnance  in  RoadRunner  '98  was  assessed  by  instructor  ratings  of  team  skills 
and  objective  measures  of  mission  accomplishment  during  the  first  and  last  missions.  Instructor 
rating  data  is  presented  only  for  F-16  pilots  and  AWACS  crews.  As  there  were  only  six  F-15 
pilots  participating  in  RoadRunner  '98,  they  could  not  form  fixed  teams  that  worked  together 
during  the  week.  The  first  and  last  missions  conducted  during  RoadRunner  '98  were  designed  to 
be  similar  and  very  demanding.  These  surface-attack  tactics  missions  required  the  F-15s  to  fly  a 
pre-strike  sweep  and  the  virtual  F-16s  to  attack  targets  located  at  an  airfield  that  was  defended  by 
both  surface  and  airborne  threats.  Bomb  score  (miss  distance)  improved  from  SAT  1  to  SAT  2 
and  losses  to  both  surface  and  airborne  threats  decreased.  The  decrease  in  red  air  kills  to  zero 
was  caused  by  a  large  reduction  in  the  number  of  engagements  between  the  virtual  F-16s  and  red 
air  threats.  Between  the  first  RoadRunner  '98  mission  and  the  last,  F-16  pilots  increased  their 
skills  at  working  with  the  AWACS  and  F-15s  to  avoid  engaging  enemy  fighters  and  to  focus 
their  resources  on  putting  bombs  on  target. 

AWACS  instructor  ratings  of  team  performance  increased  from  the  first  mission  to  the 
last  with  the  greatest  increase  in  mission  execution.  F-16  instructor  ratings  of  overall  mission 
performance,  communication,  situation  awareness,  and  crew  coordination  also  increased  after 
four  days  of  DMT  experience.  Performance  on  all  high-level  skills  assessed  using  the  TOM 
scales  improved  for  all  teams.  Review  of  each  team’s  overall  perfonnance  ratings  and  skill 
ratings  by  mission  phase,  however,  shows  greater  variability.  Team  C,  for  example, 
demonstrated  nominal  performance  for  several  mission  phases  including  brief  and  debrief  during 
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SAT  1 .  Performance  for  these  phases  did  not  increase  over  the  week  of  RoadRunner  '98  while 
teams  A  and  B  showed  considerable  improvement.  Review  of  the  TARGET  findings 
particularly  for  team  B  shows  which  specific  behaviors  were  changed  after  four  days  of  DMT . 
Team  performance  as  assessed  by  the  TARGET  scales  shows  that  experience  responding  to 
specific  scenario  events  enhanced  team  skills  such  as  coordination  between  pilots  and  AW  ACS. 
Overall,  instructor  ratings  of  team  performance  using  the  TOM/TARGET  rating  scales  are  in 
agreement  with  the  performance  measures,  questionnaire  data,  and  direct  feedback  from 
participants.  Communication  skill  was  cited  as  benefiting  from  DMT  experience  by  both  pilots 
and  AW  ACS  crews.  Other  skills  that  benefited  from  DMT  experience  were  maintaining  mutual 
support,  adhering  to  the  mission  plan,  developing  situation  awareness,  and  decision  making. 

Summary 

The  results  of  the  RoadRunner  '98  DMT  exercise  show  that  prototype  DMT  systems  are 
capable  of  providing  composite  force  training  over  a  wide  area  network  for  a  variety  of  missions. 
The  data  further  show  that  DMT  training  using  current  systems  is  most  effective  and  useful  for 
multiship,  multibogey  air  combat  with  AW  ACS  support.  DMT  training  effectiveness  can  be 
increased  for  other  missions  and  tasks  through  improvement  of  DMT  technologies,  notably 
visual  display  systems,  constructive  forces,  and  brief/debrief  systems.  Results  from  RoadRunner 
'98  also  demonstrate  that  training  effectiveness  in  DMT  can  be  assessed  through  a  combination 
of  objective  measures  of  mission  performance,  instructor  evaluations,  and  participant  feedback. 
By  examining  these  different  measures  of  effectiveness,  system  designers  and  instructors  can 
identify  the  tasks  and  skills  that  are  well  suited  to  DMT,  develop  simulator  and  instructional 
support  systems  to  improve  training  for  these  skills,  and  design  DMT  scenarios  that  are  tailored 
to  enhance  specific  warfighter  combat  skills. 


CONCLUSIONS  AND  RECOMMENDATIONS 

The  primary  conclusion  that  can  be  drawn  from  the  RoadRunner  '98  exercise  is  that 
current  DMT  technologies  are  capable  of  providing  effective,  warfighter  training  in  simulation 
that  complements  aircraft  training.  Areas  for  improvement  in  both  technologies  and  instructional 
design  were  identified  and  recommendations  are  offered  regarding  DMT  systems  and  exercises. 

Areas  for  Improvement 

No  major  shortfalls  or  showstoppers  occurred  during  RoadRunner  '98  but  many  systems 
and  strategies  were  identified  that  limited  training  effectiveness. 

DMT  systems.  Improvements  to  the  following  systems  should  result  in  significant 
increases  in  DMT  effectiveness. 

Interactions  between  entities.  Virtual  and  constructive  entities  in  RoadRunner  '98  did  not 
consistently  interact  as  they  would  in  the  real  world.  Simulation  fidelity  in  DMT  requires  that 
the  full  spectrum  of  interactions  of  offensive  and  defensive  interactions  between  players  are 
correctly  and  consistently  modeled  including  environmental  effects. 

Visual  display  systems.  High-fidelity  out-the-window  visual  display  systems  are 
required  for  all  aircraft.  These  displays  are  required  for  both  effective  training  of  team  skills 
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such  as  maintaining  tactical  formation  and  mutual  support  and  for  immersing  participants  into 
the  synthetic  environment.  For  fighter  aircraft,  visual  display  systems  must  be  able  to  support 
target  detection,  aspect  detennination,  and  recognition  at  real-world  ranges  without  introducing 
artificial  cues. 

Behavior  of  constructive  forces.  The  behavior  of  the  constructive  forces  used  in 
RoadRunner  '98  can  be  improved  in  three  areas:  flight  modeling  of  aircraft  and  missiles, 
electronic  interactions  between  entities,  and  human  behavior  models  of  pilots  and  air  defense 
crews.  Flight  models  include  speed,  range,  turning,  and  fuel  capabilities.  Electronic  interactions 
include  radar  detection  and  tracking,  radar  warning  systems,  effects  of  countermeasures,  and 
maneuvers  to  break  radar  lock.  Improving  human  behavior  models  of  enemy  warfighters  should 
include  better  modeling  of  red  force  commit/abort  criteria,  shot  doctrine,  defensive  tactics,  and 
varying  levels  of  skill  among  members  of  a  flight.  Improvements  to  behavior  models,  however, 
should  be  lower  in  priority  than  improving  flight  models  and  the  fidelity  of  electronic 
interactions.  Removing  unrealistic  behaviors  of  constructive  forces  will  have  greater  near-term 
payoff  than  improving  human  behavior  models. 

Mission  brief  and  debrief.  Effective  use  of  DMT  requires  more  than  flying  simulators 
together  over  a  wide-area  network.  Warfighters  must  be  able  to  plan,  brief,  replay,  and  debrief  a 
mission  as  individuals,  teams,  and  as  a  composite  force.  Systems  are  required  that  will  support 
virtual  interactions  among  teams  during  mission  brief  and  debrief  as  well  as  during  mission 
execution. 

Training  scenarios  and  strategies.  RoadRunner  '98  was  designed  as  an  evaluation  of  the 
potential  of  DMT  rather  than  as  a  training  exercise.  Based  on  participant  feedback  from 
RoadRunner  '98,  the  following  recommendations  can  be  made  regarding  the  design  and  conduct 
of  DMT  exercises. 

Selection  of  mission  tasks  and  skills.  The  missions,  tasks,  and  skills  to  be  trained  in  a 
DMT  exercise  should  be  selected  on  the  basis  of  three  criteria:  capabilities  of  DMT  systems  to 
support  the  proposed  training,  availability  of  aircraft  resources  for  the  proposed  training,  and  the 
anticipated  benefits  of  DMT  experience.  DMT  can  effectively  complement  aircraft  training 
when  aircraft  training  is  constrained  by  resource,  safety,  and  security  limitations.  DMT  can 
provide  experience  for  tasks  that  are  rarely  practiced  in  the  air  or,  DMT  can  provide  relevant 
experience  for  warfighters  before  they  participate  in  resource-intensive  training  events. 

Time  on  task  and  application  of  lessons  learned.  DMT  exercises  should  be  designed  to 
maximize  the  amount  of  time  spent  on  the  most  demanding  segments  of  a  mission.  Starting  a 
scenario  in  mid-flight  and  tenninating  after  training  objectives  have  been  achieved  allows  teams 
to  experience  multiple  examples  of  a  given  problem.  Further,  exercises  should  be  designed  so 
that  the  lessons  learned  in  one  DMT  session  can  be  applied  during  the  following  sessions. 

Deliberate  instruction  in  a  dynamic  environment.  DMT  is  more  than  mission  practice. 
Instructors  and  flight  leaders  must  establish  specific  training  objectives  for  a  given  team  and 
training  event.  Based  on  these  objectives,  instructors  should  identify  scenario  events  that  will 
elicit  or  demand  the  selected  skills.  DMT  designers  can  then  generate  multiple  examples  of 
scenarios  that  incorporate  these  trigger  events.  Using  this  instructional  strategy  will  allow 
instructors  to  tailor  DMT  exercises  to  meet  the  needs  of  specific  teams  and  individuals. 
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Recommendations  for  Future  Exercises 


RoadRunner  '98  was  designed  to  evaluate  and  demonstrate  the  training  potential  of  DMT 
for  composite  force  training.  Based  on  experiences  gained  from  RoadRunner  '98,  more  focused 
training  evaluations  have  been  undertaken  at  AFRL/HEA  (see  Crane,  1999;  Jensen  &  Crane, 
1999).  The  objective  of  this  research  is  to  develop  training  systems  and  strategies  for  F-16  pilots 
and  to  assess  the  impact  of  DMT  on  an  existing  training  syllabus.  In  addition  to  the  systems 
improvements  described  above,  the  following  recommendations  are  offered  to  the  developers  of 
future  composite  force,  DMT  exercises. 

Configuration  management.  A  single  agency  with  access  to  the  appropriate  models  and 
data  should  be  tasked  to  insure  that  interactions  among  all  players,  virtual  and  constructive,  are 
consistent  and  valid.  Updates  and  modifications  to  models  and  data  must  be  provided  to  all  users 
and  their  incorporation  into  DMT  exercises  assured  by  the  configuration  management  agency. 

Central  servers.  To  insure  consistency  of  effects  and  to  ease  the  burden  of  configuration 
management,  central  servers  for  weapons  modeling,  electronic  interactions,  and  environmental 
effects  should  be  incorporated  into  the  DMT  network. 

Participants  and  training  objectives.  A  major  training  payoff  of  the  RoadRunner  '98 
exercise  was  the  opportunity  for  improving  inter-team  communication  and  coordination. 
AW  ACS  controllers  and  pilots  all  benefited  from  the  experience  gained  in  RoadRunner  '98. 
Future  composite  force  DMT  exercises  could  be  designed  to  bring  together  forces  that  are 
representative  of  contemporary  air  operations  including  SEAD  elements,  force  protection, 
surface  attack,  AW  ACS,  Modular  Control  Equipment  ground  based  weapons  controllers,  Rivet 
Joint,  Unmanned  Air  Vehicle,  and  other  information  warfare  assets.  The  great  strength  of  DMT 
is  the  capability  to  complement  current  training  programs  and  large  force  exercises  with 
opportunities  for  a  wide  variety  of  warfighters  to  interact  in  an  unrestricted  environment  with 
scenarios  designed  to  exercise  specific  tasks  and  skills. 
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APPENDIX  A 


Mission  Data 


This  appendix  contains: 

1 .  The  air  order  of  battle,  and 

2.  The  following  data  for  each  of  the  seven  missions  in  RoadRunner  '98: 

•  Mission  map  including  threat  laydown  and  list  of  programmed  red  force  actions.  This 
map  was  available  to  white  cell  and  instructors  only,  not  participating  pilots  or 

AW  ACS  teams. 

•  Air  tasking  order  (ATO) 

•  Special  instructions  (SPINS) 
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Air  Order  of  Battle 


Tolicha.  lx  each  squadron  of  Flanker  and  Fulcrum  aircraft  available  for  combat  air  patrols 

(CAPs).  Typical  CAPs  include  2-ships  of  Mig-29s  or  Su-27s  to  the  south  and  4- 
ships  of  Mig-29s  to  the  north. 

Tonopah :  lx  squadron  of  Mig-23  Flogger  aircraft  in  the  ground  attack  role,  carrying  infrared 

(IR)  missiles  for  self-protection,  lx  squadron  of  Mig-29  used  periodically  in  CAPs 
to  the  east. 


Tonopah  Test  Mig-29  Fulcrum  on  10-minute  alert  at  any  time.  1-,  2-,  or  4-ships  of  Mig-29s  have 
been  seen  capping  to  the  north.  Also,  2-ships  of  Fulcrum  have  been  seen  capping  to 
the  east. 


Helen : 

Kawich : 

Korea : 

Depot  Airfield : 


Hind  helicopters  operating  in  the  resupply  /  transportation  role. 

No  operational  aircraft,  however,  it  is  capable  of  supporting  fourth-generation 
fighters. 

No  operational  aircraft. 

Transport  aircraft. 
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SAT  1  Shields  Up 


SAT  1  ATO 


// 

TASKUNIT/ 1 AVBN / KBOX 

MSNDAT/R1408/RRT1/GUNNER61/2AH64/XCAS/-/-/-/36461/ / 
MSNLOC/140800Z/140930Z/ASCOORDINATED/ALT : 0/-/3717N11548W/ / 
MSNDAT/R1409/RRT1 /HOVER63 /1AH64/XCAS /-/-/- /3  6463 / / 
MSNLOC/140800Z/ 14093 OZ/ASCOORDINATED/ALT : 0/-/3804N11604W/ / 


// 

TASKUNIT/20FW/KLSV// 

MSNDAT/A1404/RRT1/SHOOTER41/2F16C/SEAD/- /2AGM88 /-/-/3  6441/ / 
AMPN/SA-6  SITE 

TGTLOC/140810Z/140840Z/75-45NR005/5190/372454 . 1N1163134 . 3W/-/ 
75-45-05  SA-6  RDR/ / 

MSNDAT/A1405/RRT1/SHOOTER43 /2F16C/SEAD/- /2AGM88/-/-/3  6443 / / 
AMPN/SA-6  SITE 

TGTLOC/140810Z/140840Z/EC  EASTNR201/6000/373750 . 0N1162500 . 0W/-/ 
EC  EAST-201  SA-6  RDR// 


// 

TASKUNIT/27FW/KLSV/ / 

MSNDAT/A1401/RRT1/VIPER11/4F16C/INT/-/2MK84/ 6MK82/35411/ / 
AMPN/TOLICHA  AFLD 

TGTLOC/ 140 815Z/ 140 83 0Z/76-14NR2 01/53 10/3 72202 . 2N1164911 . 8W/-/ 
76-14-201  COMM  TOWER// 

TGTLOC/ 140 815Z/140 83 0Z/76-14NR2 02/53 10/372202 . INI 164 9 59 . 0W/-/ 
76-14-202  PUMP  HOUSE// 

REFUEL /EXXON7 1 /A14 11 /CALIENTEHIGH/ALT :200/140745Z/20/-/ / 
MSNDAT/A14  02 /RRT1/SNAKE21/4F16C/INT/-/2MK84/ 6MK82 /36421/ / 
AMPN/TOLICHA  AFLD 

TGTLOC/140815Z/ 140 83 0Z/76-14NR04 5/52 60/372142 . 6N1164945 . 9W/-/ 
76-14-45  RUNWAY  INT// 

TGTLOC/14 0815Z/ 140 830Z/76-14NR047/ 52 85/37214 6 . 4N1 16502 3 . 6W /-/ 
76-14-47  RUNWAY// 
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// 

TASKUNIT/3 3FW/KLSV/ / 

MSNDAT/A1406/RRTI/CYLON51/2F15C/OCA/-/BEST/-/35451// 
MSNLOC/140  800Z/140840Z/ASCOORDINATED/ALT : 0/- /3740N11500W/ / 
REFUEL/EXXON71/A1411/CALIENTEHIGH/ALT: 200/140730Z/ 40/-/ / 
MSNDAT/A1407/RRT1/CYLON53/2F15C/OCA/-/BEST/-/35453// 
MSNLOC/140800Z/140840Z/ASCOORDINATED/ALT : 0/-/3740N11500W/ / 
REFUEL/EXXON71/A1411/CALIENTEHIGH/ALT: 2 00/14073 0Z/40/-// 


// 

TASKUNIT/3 19ARW/KLSV/ / 

MSNDAT/A1411/RRT1 /EXXON71/1KC13  5/AR/-/BOM/-/3  6471 / / 
MSNLOC/140700/ 141000/CALIENTEHIGH/ALT :  200/-/3729N11422W/ / 


3 REFUEL 

/MSNNO 

/A1406 

/A1407 

/A1401 


/ACSIGN 
CYLON51 
CYLON53 
VI  PERU 


/NOTPAC 
/2F15C 
/2F15C 
/ 4F16C 


/OFF/ARCT 

/40/140730Z 

/40/140730Z 

/20/140745Z 


/TNKR/FUEL/CMNT  , 
/  -/ JP8 

/  -/ JP8 

/  -/JP8 


// 

TASKUNIT/ 5  52ACW/KLSV 

MSNDAT/A1410/RRT1/AWAKE73/1E3B/AEW/-/-/-/36473 
AMPN/ CONTROLLER  CALLSIGN : CHALISE/ / 

MSNLOC/140700Z/141400Z/CEDAR/ALT : 310 /-/3745N11320W/ / 
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Day  1  -  Tuesday  (AM) 
Pre-Test 


Note:  Some  participants  will  be  flying  SAT  1  first  and  some  will  be  flying  SAT  2  first  as 
the  pre-test.  Those  personnel  flying  SAT  1  for  the  pre-test  will  fly  SAT  2  for  the 
post-test  and  vice  versa.  The  following  information  will  display  when  there  are 
differences  between  SAT  1  and  SAT  2. 

Pre-test  Scenario  (either  SAT  1  or  SAT 2):  Day  1  of  airstrikes.  After  long  deliberations 

with  the  United  Nations,  the  president  of  the  hostile  country  has  refused  to  pull  troops 
out  of  our  ally  country.  Today  begins  a  series  of  air  strikes  designed  to  deny  the 
hostile  nation's  use  of  vital  air  assets  that  could  threaten  our  forces. 

Commander's  Intent:  My  intent  is  to  deny  the  use  of  Tolicha  airfield  by  fourth-generation 
aircraft,  specifically  Flanker  and  Fulcrum,  that  are  currently  based  out  of  Tolicha. 
These  are  high  priority  air  threats  that  may  hamper  our  efforts  in  later  airstrikes. 

Target  and  Target  Significance:  Tolicha  Airfield  communications  tower,  pump  house,  and 
runway  are  to  be  targeted  to  deny  Flanker  and  Fulcrum  aircraft  from  utilizing  support 
facilities  and  scrambling  prior  to  coalition  airstrikes. 

Operational  Surface  Threats  (SAT  1):  #1,  #3,  #5,  and  #8  are  operational.  2x  SA-8s  have 
been  active  along  the  forward  edge  of  the  battle  area  (FEBA).  Last  known  locations 
of  the  SA-8s  were  3745N  1 1615W  and  3730N  1 1615W. 

Air  Situation  (SAT  1): 

Numerous  CAPs  have  been  set  up  in  preparation  for  possible  hostilities,  those  include  both 
Mig-29s  and  Su-27  out  of  Tolicha  and  Mig-29  out  of  Tonopah  Test. 

Hind  helicopters  flying  from  Tonopah  Test  to  Tolicha  and  back  are  transporting  aircraft 
parts  and  munitions. 

2x  Hind  helicopters  flying  out  of  Helen  airfield  are  resupplying  and  transporting  troops  and 
munitions  along  the  FEBA  as  close  as  12  NM  from  the  FEBA. 

Routes  (SAT  1): 

4x  F-16CJ  (Shooter  41):  1,  3,  6.  . .  split  into  two  elements 
2-ship  F-16CJ  (Shooter  41)  against  Miller 
2-ship  F-16CJ  (Shooter  43)  against  Foster 

8x  F-16  (Viper  1 1  and  Snake  21):  1,  3,  6,  8,  11,  Target:  Tolicha,  South  egress  back  to  1 1 

4x  F-15  (Cylon  51):  Push  from  steerpoint  1  -  proceed  as  briefed  by  the  mission  commander 
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OCA  ATO 


// 

TASKUNIT7 1AVBN/KBOX 

MSNDAT/R1408/RR1B/GUNNER61/2AH64/XCAS/-/-/-/3  6461 / / 
MSNLOC/141230Z/141400Z/ASCOORDINATED/ALT: 0/-/3804N11604W/ / 


// 

TASKUNIT/ 2  OFW/KLSV/ / 

MSNDAT/A1404/RR1B/SHOOTER41/2F16C/ SEAD/-/2AGM88/-/-/36441/ / 
AMPN/SAM  SITE 

TGTLOC/141240Z / 1413 10Z /R4809 -O3NRO0 8/ 53  60/375043 . 0N1164320 .OW/-/ 
R4809-03-08  SA-2  RDR// 

MSNDAT/A1405/RR1B/SHOOTER43 /2F16C/SEAD/- /2AGM88/-/-/3  6443 / / 
AMPN/SAM  SITE 

TGTLOC/ 14 1240Z/141310Z/74-23NR0 03/53 60/372830 . 4N1161458 . 5W/-/ 
74-23-03  SA-6  RDR// 


// 

TASKUNIT/ 2  7  FW / KLSV / / 

MSNDAT/A1401/RR1B/VIPER11/ 4F16C/INT/-/2MK84/ 6MK82/35411/ / 
AMPN/MT  HELEN  AFLD 

TGTLOC/ 14 1245Z/ 1413 00Z/76-07NR010/5270/37253 7 . 0N1 1639 19 . 1W/-/ 
76-07-10  TAXIWAY  INT// 

TGTLOC /141245Z/ 1413 00Z/76-07NR011/5270/372520 . 8N1163948 . 0W/-/ 
76-07-11  RUNWAY  INT// 

TGTLOC/ 141245Z/141300Z/76-07NR01 8/52 65/3 72521 . 8N1163915 . 4W/-/ 
76-07-18  RUNWAY  INT// 

TGTLOC/141245Z/141300Z/76-07NR030/5260/372541 . 9N1163852 . 3W/-/ 
76-07-30  RUNWAY  INT 

REFUEL/EXXON71 /A1411/CALIENTEHIGH/ALT :200/141215Z/20/-/ / 
MSNDAT/A1402 /RR1B/SNAKE21/4F16C/INT/-/2MK84/ 6MK82 /3  6421/ / 
AMPN/KAWICH  AFLD 

TGTLOC/14 1245Z/ 1413  00Z/74-04NR0 08/ 5415/372412 . 7N1161432 . 4W/-/ 
74-04-08  RUNWAY// 

TGTLOC/14 1245Z/ 1413 OOZ/74-O4NR0 12/542 0/372400 . 5N1161447 . 1W/-/ 
74-04-12  RUNWAY  INT// 
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// 

TASKUNIT/33FW/KLSV// 

MSNDAT/A1406/RR1B/CYLON51/2F15C/OCA/-/BEST/-/35451/ / 

MSNLOC  / 14 12  3  OZ/ 141 3  lOZ/ASCOORDINATED/ALT : 0/- /3750N11500W/ / 
REFUEL/EXXON71/A1411/CALIENTEHIGH/ALT :200/141200Z/40/-/ / 
MSNDAT/A1407/RR1B/CYLON53/2F15C/OCA/-/BEST/-/35453// 
MSNLOC/ 14 12 3 OZ/ 14 13 lOZ/ASCOORDINATED/ALT : 0/-/3750N11500W/ / 
REFUEL /EXXON7 1 /A14 11 /CALIENTEHIGH/ ALT :200/141200Z/40/-/ / 


II 

TASKUNIT/319ARW/KLSV/ / 

MSNDAT/A1411/RR1B/EXXON71/1KC135/AR/-/BOM/-/36471// 
MSNLOC/ 14 1130/ 14143 O/CALIENTEHIGH/ALT: 200/-/3729N11422W/ / 


3 REFUEL 

/MSNNO 

/A1406 

/A1407 

/A1401 


/ACSIGN 
CYLON51 
CYLON53 
VI  PERU 


/NOTPAC 
/2F15C 
/2F15C 
/ 4F16C 


/OFF/ARCT 

/40/141200Z 

/40/141200Z 

/20/141215Z 


/ TNKR / FUEL / CMNT 
/  -/JP8 

/  -/ JP8 

/  -/JP8 


// 

TASKUNIT/ 552ACW/KLSV 

MSNDAT/A1410/RR1B/AWAKE73/1E3B/AEW/-/-/-/36473 
AMPN/ CONTROLLER  CALLSIGN : CHALISE/ / 

MSNLOC / 141 13 OZ/ 141 8 3 OZ /CEDAR/ALT : 310 /-/3745N11320W/ / 
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Day  1  -  Tuesday  (PM) 
OCA 


Commander's  Intent:  My  intent  in  afternoon  airstrikes  is  to  deny  the  use  of  Helen  and 

Kawich  airfields.  I  intend  to  render  both  Helen  and  Kawich  airfields  unusable  in  the 
event  the  hostile  nation  elects  to  use  these  locations  to  forward  deploy  strike  or  CAS 
aircraft  to  threaten  operations  along  the  FEB  A. 

Target  and  Target  Significance:  Helen  and  Kawich  airfield  DMPIs  include  runways, 
taxiways,  and  runway  intersections. 

BDA:  Surface  threats  #3  and  #5  were  suppressed  by  F-16  SEAD  assets  during  the  morning 
airstrikes.  Threat  #8  is  unlocated  -  it  is  assessed  to  be  in  travel  mode. 

Operational  Surface  Threats:  #1  remains  operational  from  earlier  in  the  day.  #6  is  a  new, 
high  priority  SA-6  threat  in  the  vicinity  of  the  FEBA.  2x  SA-8s  have  been  identified 
as  operational  along  the  FEBA.  The  last  known  locations  of  the  SA-8s  were  3745N 
11615W  and  3730N  11615W. 

Air  Situation: 

2-ship  Mig-29  caps  active  out  of  Helen  and  Tonopah  airfields. 

2  Hind  helicopters  continue  to  conduct  resupply  and  transportation  of  troops  and  munitions 
out  of  Helen  airfield. 

Routes: 

4x  F-16CJ  (Shooter  41):  1,  IP  (Belted  Peak) . . .  split  into  two  elements  against  Falstaff  and 
Amstel 

4x  F-16C  (Viper  1 1):  2,  IP  is  7,  Tgt:  Helen  Airfield,  egress  south  to  steerpoint  8,  egress  east 

4x  F-16C  (Snake  21):  1,  IP  (Belted  Peak),  Tgt:  Kawich  Airfield,  egress  south  and  east 

4x  F-15  (Cylon  5 1):  Push  from  Steerpoint  2,  then  proceed  as  mission  commander  directs 
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INTERDICTION  ATO 


// 

TASKUNIT/ 1 AVBN / KBOX 

MSNDAT/R1508/RR2A/GUNNER61/2AH64/XCAS/-/-/-/36461// 
MSNLOC/150800Z/150930Z/ASCOORDINATED/ALT : 0/-/3722N11512W/ / 
MSNDAT/R1509/RR2A/HOVER45/1AH64/XCAS/-/-/-/3  6445 / / 

MSNLOC/ 15 0800Z/ 15093 OZ/ASCOORDINATED/ALT : 0/-/3804N11604W/ / 


II 

TASKUNIT/20FW/KLSV/ / 

MSNDAT/A1504/RR2A/SHOOTER41/2F16C/SEAD/-/2AGM88/-/-/3  6441 / / 
AMPN/SAM  SITE 

TGTLOC/150810Z/150840Z/R4809-03NR008/5300/375043 . ONI 164320 . 0W /-/ 
R4809-03-08  SA-2  RDR// 

MSNDAT/A1505/RR2A/SHOOTER43 /2F16C/SEAD/-/2AGM88/-/-/3  6443 / / 
AMPN/SAM  SITE 

TGTLOC/150810Z/150840Z/74-23NR003/5360/372830 . 4N1161458 . 5W/-/ 
74-23-03  SA-6  RDR// 


// 

TASKUNIT/27FW/KLSV// 

MSNDAT/A1501/RR2A/SNAKE11/ 4F16C/INT/-/ 6MK82/2MK84/36411/ / 
AMPN/MT  HELEN  AFLD 

TGTLOC/ 15 0 815Z /150 83 0Z/ 7 6-07NR0 02/52 80/37253 3 . 3N1163949 . 6W/-/ 
76-07-02  BUNKER// 

TGTLOC/ 15 0815Z/ 150 83 0Z/76-07NR003/5270/3 72550 . 8N1163849 .7W/-/ 
76-07-03  BUNKER// 

MSNDAT/A1502/RR2A/VIPER21/4F16C/INT/-/ 6MK82/2MK84/35421/ / 

AMPN/ STORAGE  DEPOT 

TGTLOC/ 15 0 815Z/ 150 83 0Z/R4809-01NR001/ 56 00/373732 . 1N11 64145 . 0W/-/ 
R4809-01-01  BLDG// 

TGTLOC/ 15 0815Z/150 83 0Z/R4809-01NR002 / 56 00/37 3729. 3N1164144 . 2W/-/ 
R4809-01-02  BLDG// 

TGTLOC/ 15 0815Z/ 150 83 0Z /R4809 -01NR003 / 559 0/37372 6 . 7N1164143 . 7W/-/ 
R4 809-01-03  BLDG// 

TGTLOC/150 815Z/ 15083 OZ/R4809-01NR005/55 9 0/37 3727 .2N11 64148 .7W/-/ 

R4809-01-05  BLDG// 

REFUEL / EXXON7 1 / A1 511 /CALIENTEHIGH/ ALT :200/150745Z/20/-/ / 
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// 

TASKUNIT/ 3  3FW/KLSV/ / 

MSNDAT/A1506/RR2A/CYLON51/2F15C/OCA/-/BEST7-/35451// 
MSNLOC/152100Z/152140Z/ASCOORDINATED/ALT : 0/-/3750N11500W/ / 
REFUEL/EXXON71/A1511/CALIENTEHIGH/ALT: 2 00/15073 0Z/40/-// 
MSNDAT/A1507/RR2A/CYLON53/2F15C/OCA/-/BEST/-/35453// 
MSNLOC/152 100Z/152140Z/ASCOORDINATED/ALT : 0/-/3750N11500W/ / 
REFUEL/EXXON71/A1511/CALIENTEHIGH/ALT: 2 00/15073 0Z/40/-// 


// 

TASKUNIT/ 3 19ARW/KLSV/ / 

MSNDAT/A1511/RR2A/EXXON71/1KC135/AR/-/BOM/-/36471// 
MSNLOC/152 000/152200/CALIENTEHIGH/ALT : 200/-/3729N11422W/ / 
3 REFUEL 


/MSNNO 

/ACSIGN 

/NOTPAC 

/OFF/ARCT 

/ TNKR / FUE L / CMNT 

/A150 6 

CYLON51 

/2F15C 

/40/150730Z 

/ 

-/JP8 

/A1507 

CYLON53 

/2F15C 

/40/150730Z 

/ 

-/ JP8 

/A1502 

VIPER21 

/ 4F16C 

/20/150745Z 

/ 

-/JP8 

// 

TASKUNIT/ 5  52ACW/KLSV 

MSNDAT/A1510/RR2A/AWAKE73 /1E3B/AEW/-/-/-/36473 
AMPN/ CONTROLLER  CALLSIGN : CHALISE/ / 

MSNLOC/150700Z/ 1514 00Z /CEDAR/ALT: 31 0/-/3745N1 132 0W// 
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Day  2  -  Wednesday  (AM) 
Interdiction 


Scenario:  During  a  temporary  cessation  of  coalition  airstrikes  in  the  morning,  the  enemy 
took  the  opportunity  to  send  strike  aircraft  to  attack  our  forward  operating  bases. 
Coalition  forces  will  resume  offensive  action  toward  the  hostile  nation. 

Commander’s  Intent:  My  intent  today  is  to  remove  the  enemy  ground  troop's  ability  to 
resupply  with  weapons  and  munitions  and  repair  with  spare  parts.  Also,  with  the 
previous  day's  airstrikes  to  airfields  considered  vital  to  the  enemy  nation,  I  will  also 
remove  their  capability  to  repair  damaged  aircraft  with  spare  parts. 

Target  and  Target  Significance:  Targets  include  buildings  at  a  storage  depot  and  bunkers  at 
Helen  Airfield.  Buildings  in  the  storage  depot  will  be  targeted  to  deny  the  enemy  the 
ability  to  resupply  army  troops.  Bunkers  at  Helen  Airfield  will  be  targeted  to  deny 
the  enemy  the  use  of  munitions,  which  we  suspect  supports  ground  troops. 

BDA:  #4  was  effectively  destroyed  by  SEAD  assets.  #1  remains  active. 

Operational  Surface  Threats:  #1,  #2,  #6,  and  #7  are  operational.  3x  SA-8s  have  been 

active  along  the  FEBA.  The  last  known  locations  of  the  SA-8s  were  3745N  1 1615W, 
3730N  1 1615W,  and  3715N  11615W. 

Air  Situation: 

4-ship  cap  of  Mig-29  south  of  Tonopah 

2x  2-ship  caps  of  Mig-29  capping  south  of  Tolicha 

2x  Hind  aircraft  operating  between  Tonopah  and  Helen  airfields  are  running  resupply  flights 
to  two  points  along  the  FEBA:  03752N  1 1620W  and  03730N  1 1620W. 

Routes: 

4x  F-16CJ  (Shooter  41)  push  from  Steerpoint  1  . . .  splits  into  two  elements 
2x  F-16  CJ  (Shooter  41)  1,5,  target  Amstel 

2x  F-16  CJ  (Shooter  43)  1,  3,  9  target  Stroh’s  (set  up  between  Strohs  and  Lonestar) 

4x  F-16C  (Snake  11)1,3,9,  Target:  Helen  -  egress  North  and  East  to  Steerpoint  9 

4x  F-16C  (Viper  21)  1,  5,  7, 10,  Target:  Storage  Depot  -  egress  South  to  Steerpoint  9 

4x  F-15  (Cylon  51)  Push  from  Steerpoint  2  then  as  directed  by  mission  commander 


64 


DCA  Shields  Up 


4  x  F-1 5  proceeding  North  DCA  CAP  :09  N  4  x  MiG-23  strikers  takeoff  Tonopah  :34  N  4  x  MiG-23  strikers  takeoff  North  AFLD 

4  x  F-16  South  DCA  CAP  :09  N  2  x  MiG-29  (Center  CAP)  maneuver  :36  N  2  x  MiG-29  (Man)  takeoff  Tonopah 

:1 1  N  2  x  MiG-29  (Center  CAP)  maneuver  :42  R  2  x  MiG-29  (North  CAP)  attack 

:1 8  fl  2  x  MiG-29  (Man)  attack  :44  R  2  x  Su*27  (South  CAP)  attack 

:23  N  2  x  Su-27  (South  CAP)  attack/drag 


DCA  ATO 


// 

TASKUNIT/27FW/KLSV/ / 

MSNDAT/A1501/RR2B/ SNAKE11/ 4F16C/DCA/-/BEST/- /3  6411/ / 
MSNLOC/151140Z/151240Z/ASCOORDXNATED/ALT: 0/-/3725N11525W// 
REFUEL / EXXON7 1 / A1 511 /CALIENTEHIGH/ ALT :200/151115Z/20/-/ / 
MSNDAT/A1502/RR2B/VIPER21/4F16C/DCA/-/BEST/-/35421// 
MSNLOC/151240Z/151340Z/ASCOORDINATED/ALT : 0/-/3725N11525W/ / 
REFUEL/EXXON71/A1511/CALIENTEHIGH/ALT :200/151215Z/20/-/ / 


// 

TASKUNIT/33FW/KLSV/ / 

MSNDAT/A150 6 /RR2B/CYLON51/2F15C/OCA/- /BEST/-/ 35451/ / 
MSNLOC/151230Z/151330Z/ASCOORDINATED/ALT: 0/-/3755N11525W/ / 
REFUEL/EXXON71/A1511/CALIENTEHIGH/ALT :200/1512000Z/40/-// 
MSNDAT/A1507/RR2B/CYLON53/2F15C/OCA/-/BEST/-/35453// 
MSNLOC/151230Z/151330Z/ASCOORDINATED/ALT : 0/-/3755N11525W/ / 
REFUEL/EXXON71/A1511/CALIENTEHIGH/ALT: 200/151200Z/ 40/-/ / 


// 

TASKUNIT/319ARW/KLSV// 

MSNDAT/A1511/RR1A/EXXON71/1KC135/AR/-/BOM/-/36471/ / 
MSNLOC/151100/151400/CALIENTEHIGH/ALT : 200/-/3729N11422W/ / 


4REFUEL 

/MSNNO 

/A1501 

/A1506 

/A1507 

/A1502 


/ACSIGN 
SNAKE 11 
CYLON51 
CYLON53 
VIPER21 


/NOTPAC 

/4F16C 

/2F15C 

/2F15C 

/4F16C 


/OFF/ARCT 

/20/151115Z 

/40/151200Z 

/40/151200Z 

/20/151215Z 


/ TNKR / FUEL / CMNT 
/  -/JP8 

/  -/JP8 

/  -/JP8 

/  -/JP8 


// 

TASKUNIT/ 5  52ACW/KLSV 

MSNDAT/A1510/RR1B/AWAKE73/1E3B/AEW/-/-/-/36473 
AMPN/ CONTROLLER  CALLSIGN : CHALISE/ / 

MSNLOC/151030Z/151730Z/CEDAR/ALT : 310 /-/3745N11320W/ / 
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Day  2  -  Wednesday  (PM) 
DCA 


Scenario:  In  an  attempt  to  allow  the  hostile  nation  the  opportunity  to  withdraw  military 

forces  and  cease  offensive  operations,  the  coalition  partners  have  agreed  to  postpone 
airstrikes  in  favor  of  a  defensive  posture. 

Commander's  Intent:  I  intend  to  conduct  defensive  operations  in  an  effort  to  stop  an  enemy 
attack  against  our  oil-producing  ally.  By  transitioning  to  a  defensive  posture,  I  intend 
to  allow  the  hostile  nation  the  opportunity  to  take  the  deciding  step  toward  a  peaceful 
resolution  or  a  resumption  of  coalition  airstrikes. 

Target  and  Target  Significance:  N/A 

BDA:  #1  and  #2  are  unlocated. 

Operational  Surface  Threats:  #5,  #6,  and  #7  are  new  operational  threats.  SA-8  activity  has 
been  noted  along  the  FEBA.  The  last  known  locations  of  the  SA-8s  were  3745N 
11615W  and  3730N  11615W. 

Air  Situation: 

3-ship  of  Su-27  and  2-ship  of  Mig-29  out  of  Tolicha 

Note:  Lanes  change  for  F-15/16  operating  areas.  F-15  are  north  of  3740N  and  F-16  are 
south  of  3740N. 


67 


SEAD-C  Shields  Up 


i-24  N  Target  Area  :14  fvl  2  x  MiG-29  (Center  CAP)  attack  Depot  Airfield  (2 TAG) 

:1 5  INI  2  x  Su-27  takeoff Tolicha  for  South  CAP 

:16  N  AH-64  takeoff  Alamo 

:22  INI  2  x  MiG-29  (Man)  takeoff  Ton opah 


SEAD  ATO 


// 

TASKUNIT/ 1AVBN/KBOX 

MSNDAT/R1608/RR3A/GUNNER61/2AH64/XCAS/-/-/-/36461// 

MSNLOC /160800Z/1609  3  OZ /ASCOORDINATED/ALT : 0 / - / 3  722N1 15 12W/ / 
MSNDAT/R1609/RR3A/HOVER45/1AH64/XCAS/-/-/-/36445// 
MSNLOC/160 800Z/ 16093 OZ /ASCOORDINATED/ALT : 0/-/ 3 804N11604W// 


// 

TASKUNIT / 20FW/KLSV/ / 

MSNDAT/A1604/RR3A/SHOOTER41/2F16C/SEAD/-/2AGM88/-/-/3  6441 / / 
AMPN/SA-6 

TGTLOC/160810Z/ 161 840Z/REVEILLE-01NR001/ 6 100/375750. 0N11 61950 . OW/-/ 
REVEILLE-01-01  SA-6  RDR// 


// 

TASKUNIT/27FW/KLSV/ / 

MSNDAT/A1601/RR3A/VIPER11/4F16C/INT/-/CBU87 / 6MK82/35411/ / 
AMPN/SA-2 

TGTLOC/160815Z/160830Z/76-10NR007/5225/372402 . 5N1165204 . 8W/-/ 
76-10-07  SA-2 / / 

TGTLOC/ 160 815Z / 160 83 0Z/76-10NR003/ 5225/3 724 06 . 9N1165205 . 5W/-/ 
76-10-03  SA-2// 

TGTLOC/ 160 815Z/1 6083 0Z/76-10NR00 6/ 5225/3 72403 . 9N11652 01 . 7W/-/ 
76-10-06  SA-2// 

TGTLOC/ 160 815Z/ 16083 0Z/76-10NR004/5225/3 72407 .9N11652 03 .9W/-/ 
76-10-04  SA-2// 

REFUEL/EXXON71/A1611/CALIENTEHIGH/ALT :200/160745Z/20/-/ / 

MSNDAT / Al 602 /RR3 A/ SNAKE2 1/4F16C/ INT/ - / 2MK84 / 6MK8  2/36421/ / 

AMPN/ KOREAN  AFLD 

TGTLOC/160815Z/160830Z/71-05NR002/4 810/3 73527 . 0N1 165 527 . 5W/-/ 
71-05-02  RUNWAY  INT// 

TGTLOC/160815Z/160830Z/71-05NR025/4830/37353  8 . 4N1165505 . 8W/-/ 
71-05-25  FUEL  STORAGE// 

MSNDAT/A1603/RR3A/DILLER31/2F16C/INT/-/2MK84/ 6MK82/36431/ / 
AMPN/DEPOT  AFLD 

TGTLOC/1 60 815Z/ 16083 0Z/R480 9 -02NROO 1/53 60/373 832 . 6N1163855 . 5W/-/ 
R4809-02-01  RUNWAY  INT// 

TGTLOC/160815Z/160830Z/R4809-02NR002 /5340/373 916 . 0N1163910 . 3W/-/ 
R4809-02-02  PARKING  RAMP// 
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// 

TASKUNIT/33FW/KLSV// 

MSNDAT/A1 606 /RR3A/CYLON51/2F15C/OCA/ -/BEST/- /35451/ / 
MSNLOC/160800Z/160840Z/ASCOORDINATED/ALT : 0/-/3737N11500W/ / 
REFUEL / EXXON7 1 / Al 6 1 1 /CALIENTEHIGH/ ALT :200/160730Z/40/-// 
MSNDAT/A1607/RR3A/CYLON53/2F15C/OCA/-/BEST/-/35453// 
MSNLOC/160800Z/160840Z/ASCOORDINATED/ALT : 0/-/3737N11500W/ / 
REFUEL/EXXON71/A1611 /CALIENTEHIGH/ ALT : 2 00/16073 OZ/40/-/ / 


// 

TASKUNIT/ 3 1 9ARW/KLSV/ / 

MSNDAT/A1611/RR3A/EXXON71/1KC135/AR/-/BOM/-/36471// 
MSNLOC/161600/161900/CALIENTEHIGH/ALT: 200/-/3729N11422W/ / 
3 REFUEL 


/MSNNO 

/ACSIGN 

/NOTPAC 

/OFF/ARCT 

/ TNKR /FUEL/ CMNT 

/A1606 

CYLON51 

/2F15C 

/40/160730Z 

/ 

-/ JP8 

/A1607 

CYLON53 

/2F15C 

/40/160730Z 

/ 

-/ JP8 

/A1601 

VI  PERU 

/4F16C 

/20/160745Z 

/ 

-/JP8 

// 

TASKUNIT/ 5  52ACW/KLSV 

MSNDAT/A1610/RR3A/AWAKE73/1E3B/AEW/-/-/-/36473 
AMPN/ CONTROLLER  CALLSIGN : CHALISE// 

MSNLOC/160700Z/161400Z/CEDAR/ALT : 310/-/3745N11320W/ / 


70 


Day  3  -  Thursday  (AM) 
SEAD-C 


Scenario:  Airstrikes  continue  with  no  change  to  the  air  situation.  Ground  troops  have  been 
observed  massing  along  the  central  portion  of  the  FEBA. 

Commander's  Intent:  Due  to  the  abundance  of  active  SAMs  operating  in  the  area,  I  intend 
to  suppress  air  defense  assets  to  open  up  a  corridor  for  coalition  strike  aircraft.  I  also 
intend  to  deny  the  use  of  both  Korea  Airfield  and  Depot  Airfield,  as  the  options  for 
the  hostile  nation  are  narrowing  for  forward  deploying  enemy  aircraft. 

Target  and  Target  Significance:  Strikers  will  bomb  the  SA-2  site  to  support  the  SEAD 
objective.  Korean  Airfield  runway  and  fuel  storage,  and  the  Depot  Airfield  runway 
intersection  and  parking  ramp  will  be  targeted  to  render  them  unusable  for  future 
forward  deployment  of  enemy  assets. 

BDA:  Neither  threats  #1  nor  #6  surface  threats  were  destroyed  by  SEAD  assets  on 
Wednesday  afternoon. 

Operational  Surface  Threats:  #1,  #3,  #6,  and  #7  are  operational.  2x  SA-8s  are  operational 
along  the  FEBA.  The  last  known  locations  of  the  SA-8  were  3745N  1 1615  W  and 
3715N  11615W. 

Air  Situation: 

2x  Su-27  capping  south  of  Tolicha 

2x  Mig-29  capping  along  3730  between  Helen  and  Depot  Airfield 

Hinds  operating  in  the  vicinity  of  the  FEBA,  flying  a  similar  profile  as  yesterday. 

Routes: 

4x  F-16CJ  (Shooter  41)  2,  4,  5,  target  Amstel 

4x  F-16C  (Viper  1 1)  1,  3,  9, 12,  Target:  Lonestar  -  egress  North  to  Steerpoint  12  then  9 

4x  F-16C  (Snake  21)  2, 4,  10, . .  .  then  break  into  two  elements 

(Snake  21)  2,  4,  10, 12,  Target:  Korean  Airfield  -  egress  South  to  Steerpoint  12  then  9 

(Snake  23)  2,  4,  10,  Target:  Depot  Airfield  -  egress  South  to  9 

4x  F-15  (Cylon  51)  Push  from  Steerpoint  1  then  as  directed  by  mission  commander 
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CAS  Shields  Up 


>  identifies  unknown  SA-8 

-1iG*29  (Man)  takeoff  Tonopah 

Armored  column  appears  South  Kawich  Valley 


CAS  ATO 


// 

TASKUNIT/ 1AVBN/KBOX 

MSNDAT/R1608/RR3B/GUNNER61/2AH64/XCAS/-/-/-/36461// 
MSNLOC/161230Z/161400Z/ASCOORDINATED/ALT : 0/-/3722N11512W/ / 
MSNDAT/R1609/RR3B/HOVER45/1AH64/XCAS/-/-/-/36445// 
MSNLOC/161230Z/161400Z/ASCOORDINATED/ALT: 0/-/3804N11604W// 


// 

TASKUNIT/27FW/KLSV/ / 

MSNDAT/A1601/RR3B/SNAKE11/ 4F16C/INT/-/2MK84/ 6MK82 /3  6411/ / 
AMPN/STORAGE  DEPOT/AIRFIELD 

TGTLOC/161245Z/ 1613 00Z/R48 09 -OINROO 1/5600/373 732 . 1N1164145 . OW/-/ 
R4809-01-01  BLDG// 

TGTLOC/ 16 1245Z/ 1613 00Z/R48 09 -01NR004/ 5600/373 730 . 4N11 64149 . 3W/-/ 
R4809-01-04  BLDG// 

TGTLOC/ 16 1245Z/161300Z/R4809-02NR001/ 53 6 0/373 832 . 6N1163855 . 5W/-/ 
R4809-02-01  RUNWAY  INT// 

TGTLOC/ 161245Z/161300Z/R4809-02NR002/53 40/373 91 6 . 0N11 63910 . 3W/-/ 
R4809-02-02  PARKING  RAMP// 

MSNDAT/A1602/RR3B/DILLER21/4F16C/INT/-/CBU87 / 6MK82/36421/ / 
AMPN/SA-2 

TGTLOC/161245Z/ 1613 00Z/R48 09 -O3NRO0 1/53 00/375045 . 0N1 16 43 3 3 . 7W/-/ 
R4809-03-01  SA-2/ / 

TGTLOC / 161245Z/ 1613 00Z/R4809-03NR00 6/ 530 0/37 5045 . 0N11643319 . 3W/-/ 
R4809-03-06  SA-2// 

TGTLOC/161245Z/1613  00Z/R4809-03NR003 /5300/375041 . 0N116433  8 . 5W/-/ 
R4809-03-03  SA-2// 

TGTLOC/ 161245Z/ 1613 00Z/R48 09 -03NR004/53 00/375041 . 1N1164339 . 8W/-/ 
R4809-03-04  SA-2// 

MSNDAT/A1603 /RR3B/VIPER3 1/4F16C/CAS/-/CBU87 / 6MK82 /35431/ / 
AMPN/ARMOR 

MSNLOC/161245Z/161330Z/KILLBOX  NB-08/-/-/ 
3720N11620W/3740N11620W/3740N11600W/3720N11600W// 

FACXNFO/HAWG01 /RED  06 /ORANGE  03/SP-03/-/-/ 

REFUEL /EXXON7 1 /A1 6 1 1/CALIENTEHIGH/ALT :200/161215Z/20/-// 


// 

TASKUNIT / 33FW/KLSV/ / 

MSNDAT/A1606/RR3B/CYLON51/2F15C/DCA/-/-/-/35451// 

MSNLOC/ 16123 0Z/ 16133 OZ/ASCOORDINATED/ALT : 0/-/3750N1153  5W/ / 
REFUEL/EXXON7 1 / A1 611 /CALIENTEHIGH/ALT :200/161200Z/40/-/ / 
MSNDAT / A1 6  0  7 / RR3  B / C YLON 53/2F15C/ DCA /-/-/-/35453/ / 
MSNLOC/161230Z/ 1613 3 OZ/ASCOORDINATED/ALT : 0/- /3750N11535W/ / 
REFUEL /EXXON7 1 / A1 611 /CALIENTEHIGH/ALT :200/161200Z/40/-// 


// 

TASKUNIT/ 3 54FW/KLMO 

MSNDAT/A1612/RR3B/HAWG01/10A-10/XCAS/-/-/-/35401// 
MSNLOC/161230Z/ 1613 3 OZ/ASCOORDINATED/ALT : 0/- /3730N11600W/ / 
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// 

TASKUNIT/ 3 1 9ARW/KLSV/ / 

MSNDAT/A1611/RR3B/EXXON71/1KC135/AR/-/BOM/-/36471/ / 
MSNLOC/161130/161430/CALIENTEHIGH/ALT: 200/-/3729N11422W/ / 
3 REFUEL 


/MSNNO 

/ACSIGN 

/NOTPAC 

/OFF/ARCT 

/ TNKR / FUEL / CMNT 

/A1606 

CYLON51 

/2F15C 

/40/161200Z 

/ 

-/ JP8 

/A1607 

CYLON53 

/2F15C 

/40/161200Z 

/ 

-/JP8 

/A1603 

VIPER31 

/4F16C 

/20/161215Z 

/ 

-/JP8 

II 

TASKUNIT/ 5  52ACW/KLSV 

MSNDAT/A1610/RR3B/AWAKE73/1E3B/AEW/-/-/-/36473 
AMPN/ CONTROLLER  CALLSIGN : CHALISE/ / 

MSNLOC/161130Z/161830Z/CEDAR/ALT : 310 /-/3745N11320W/ / 
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Day  3  -  Thursday  (PM) 
CAS 


Scenario:  The  massing  of  enemy  Armor,  Mechanized  Infantry,  and  Attack  Helicopters 

advancing  towards  and  in  the  central  portion  of  the  FLOT/FEB  A  indicates  the  enemy 
has  pushed  over  the  FLOT/FEBA. 

Commander’s  Intent:  I  intend  to  utilize  CAS  assets  in  the  Kill  Box  and  in  the  central  area  of 
the  FLOT/FEBA  to  destroy  the  enemy’s  foothold  and  discourage  any  further  enemy 
offensive  attacks.  US  ground  forces  will  remain  in  a  defensive  posture  conducting 
Defensive  Security  Operations  and  will  not  engage  enemy  air  attacks  unless 
authorized  to  fire  by  the  Fire  Support  Element  (FSE)  and  Tactical  Operations 
Command  (TOC).  US  ground  forces  will  defend  themselves  with  Air  Defense 
Artillery  (ADA)  and  other  measures  as  directed.  Rules  of  engagement  will  be  Visual 
Recognition  of  Enemy  Aircraft  before  engaging. 

Target  and  Target  Significance:  The  storage  depot  support  buildings,  runway  intersection, 
and  parking  ramp.  SA-2  site  and  Kill  box  area,  concentrating  on  enemy  main  battle 
tanks  and  enemy  ADA. 

BDA:  NSTR 

Operational  Surface  Threats:  #3  and  #8  are  active.  #1,  #6,  and  #7  are  unlocated.  3x  SA-8s 
have  been  active  along  the  FEBA.  The  last  known  locations  of  the  SA-8s  were 
3745N  11615W,  3730N  11615W,  and  3715N  11615W. 

Air  Situation: 

2x  Hind  have  been  operating  in  the  vicinity  of  the  FEBA 

Some  Flogger  activity  has  been  noted  at  Tonopah  Airfield 

Routes: 

4x  F-16CJ  (Snake  11)1,4,  10,  Target:  Depot  Airfield  -  egress  South 

4x  F-l 6C  (Diller  21)  2,  5,  7,  Target:  Amstel  -  egress  South 

4x  F-l 6C  (Viper  31) 

Contact  Point:  Steerpoint  1 
Initial  Point:  Steerpoint  3 

lx  A- 10  (Hog  01)  will  be  on  station  capping  southwest  of  Steerpoint  3 

4x  F-l 5  (Cylon  51)  pushing  from  Steerpoint  2 
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SAT  2  ATO 


// 

TASKUNIT/ 1AVBN/KBOX 

MSNDAT/R14 0 8 /RRT2 /GUNNER6 1 / 2 AH6 4 /XCAS/ - / - / - / 3 64 6 1 / / 
MSNLOC/140800Z/140930Z/ASCOORDINATED/ALT: 0/-/3717N11548W/ / 
MSNDAT/R1409/RRT2/HOVER63/1AH64/XCAS/-/-/-/36463// 
MSNLOC/140800Z/140930Z/ASCQORDINATED/ALT: 0/-/3804N11604W// 


// 

TASKUNIT/20FW/KLSV/ / 

MSNDAT/A1404/RRT2/SHOOTER41 / 2F16C/ SEAD/-/2AGM88/- /-/3  6441 / / 
AMPN/SA-6  SITE 

TGTLOC/140810Z/ 140 840Z/75-45NR0 05/5 190/372454 . 1N1163134 . 3W/-/ 
75-45-05  SA-6  RDR/ / 

MSNDAT/A1405/RRT2 /SHOOTER43 /2F16C/SEAD/-/2AGM88/- /-/3  6443 / / 
AMPN/SA-6  SITE 

TGTLOC/140810Z/140840Z/EC  EASTNR201/6000/373750 . 0N1162500 . 0W/-/ 
EC  EAST-201  SA-6  RDR// 


// 

TASKUNIT/27FW/KLSV// 

MSNDAT/A1401/RRT2/VIPER11/4F16C/INT/-/2MK84/ 6MK82/35411/ / 
AMPN / TOLICHA  AFLD 

TGTLOC/140815Z/ 140830Z/76-14NR201/5310/372202 . 2N1164911 . 8W/-/ 
76-14-201  COMM  TOWER// 

TGTLOC/ 140 815Z/ 14083 0Z/76-14NR2  02/53 10/3 72202 . INI 1649 5 9 . 0W/-/ 
76-14-202  PUMP  HOUSE// 

REFUEL /EXXON7 1 /A14 11 /CALIENTEHIGH/ALT : 200/1407 45Z/ 20/-/ / 
MSNDAT/A1402/RRT2/SNAKE21/4F16C/INT/-/2MK84/ 6MK82 /36421/ / 
AMPN /TOLICHA  AFLD 

TGTLOC/140815Z/ 140 83 0Z/76-14NR045/ 52 60/372142 . 6N1164945 . 9W/-/ 
76-14-45  RUNWAY  INT// 

TGTLOC/ 14 0815Z/ 140 83 0Z/ 7 6-14NR047/52  85/372146 . 4N1165023 . 6W/-/ 
76-14-47  RUNWAY// 
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// 

TASKUNIT/33FW/KLSV// 

MSNDAT/A1406/RRT2/CYLON51/2F15C/OCA/-/BEST/-/35451// 
MSNLOC/140800Z/140840Z/ASCOORDINATED/ALT: 0/-/3740N11500W// 
REFUEL/EXXON71/A1411/CALIENTEHIGH/ALT : 200 / 14073 OZ/ 40/-/ / 
MSNDAT/A1407/RRT2 /CYLON53 /2F15C/OCA/- /BEST/- /3 5453 / / 
MSNLOC/ 14 0 800Z/140840 Z / ASCOORDINATED/ ALT :  0 / - /  37 4 ONI  1 50 OW / / 
REFUEL/EXXON7 1 / A14 11 /CALIENTEHIGH/ALT :200/140730Z/40/-// 


// 

TASKUNIT / 3 1 9ARW/KLSV/ / 

MSNDAT/A1411 /RRT2 /EXXON71/1KC135/AR/-/BOM/-/36471/ / 
MSNLOC/ 14 07 00/141 000 /CALIENTEHIGH/ALT : 200 /-/3729N11422W/ / 
3REFUEL 


/MSNNO 

/ACSIGN 

/NOTPAC 

/OFF/ARCT 

/ TNKR / FUEL / CMNT 

/A1406 
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Day  4  -  Friday 
Post-Test 


Note:  Some  participants  will  be  flying  SAT  1  first  and  some  will  be  flying  SAT  2  first  as 

the  post-test.  Those  personnel  who  flew  SAT  1  for  the  pre-test  will  fly  SAT  2  for  the 
post-test  and  vice  versa.  The  following  information  will  display  when  there  are 
differences  between  SAT  1  and  SAT  2  for  the  post-test  scenario. 

Post-test  Scenario  (either  SAT  1  or  SAT  2) :  Given  the  tenacity  of  the  enemy  forces,  the 
president  of  the  hostile  nation  has  decided  to  provide  a  last-ditch  effort  in  repairing 
runways  and  attempting  to  launch  aircraft  out  of  Tolicha  airfield. 

Commander's  Intent:  My  intent  is  to  deny  the  use  of  Tolicha  airfield  by  fourth-generation 
aircraft,  specifically  Flanker  and  Fulcrum,  that  are  currently  based  out  of  Tolicha. 

Target  and  Target  Significance:  Tolicha  Airfield  communications  tower,  pump  house,  and 
runway  are  to  be  targeted  to  deny  Flanker  and  Fulcrum  aircraft  from  utilizing  support 
facilities  and  scrambling  during  coalition  airstrikes. 

BDA:  #1  has  not  been  destroyed  and  remains  active. 

Operational  Surface  Threats  (SAT  2):  #1,  #5,  #6,  and  #8  are  operational.  2x  SA-8s  are 
operational  along  the  FEB  A. 

Air  Situation  (SAT  2): 

2x  Mig-29  are  capping  out  of  Tonopah 

4-ship  and  a  2-ship  of  Mig-29  are  capping  out  of  Tolicha 

2x  Hind  are  operating  in  the  vicinity  of  the  FEB  A  and  lx  Hind  is  flying  a  route  to/from 
Tonopah  and  Tolicha,  moving  aircraft  parts. 

Routes  (SAT 2): 

4x  F-16CJ  (Shooter  41)  push  from  Steerpoint  1,4...  splits  into  two  elements 
2x  F-16  CJ  (Shooter  41)  1, 4,  9,  Target  threats  between  Pils  and  Strohs 
2x  F-16  CJ  (Shooter  43)  1, 4,  Target  threats  between  Amstel  and  Miller 

9x  F-16C  (Snake  21  and  Viper  1 1)  1, 4,  9,  12,  Target:  DMPIs  on  Tolicha  Airfield  -  egress 
South  to  Steerpoint  9 

4x  F-15  (Cylon  51)  Push  from  Steerpoint  2  then  as  directed  by  mission  commander 
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APPENDIX  B 


F-16  Air-to-Air  Scenarios 

This  appendix  contains  depictions  of  the  air-to-air  scenarios  conducted  by  F-16  teams  at 
AFRL/HEA  after  completion  of  RoadRunner  '98  missions. 
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Abstract 


Distributed  Mission  Training  (DMT)  enables  participants  to  perform  within  a  virtual  battlespace 
created  through  networking  of  several  high-fidelity  simulations.  In  a  recent  exercise, 
RoadRunner  '98,  several  agencies  interacted  to  create  several  battlespace  missions  in  which 
“friendly”  fighter  aircraft  and  command  and  control  crewmembers  participated  as  trainees,  while 
supporting  roles  and  enemy  forces  were  either  played  by  operational  personnel  (virtual  players) 
or  created  by  intelligent  agent  technology  (constructed  forces).  Thus,  trainees  participated  in 
complex  demanding  war  scenarios  without  the  usual  constraints  of  cost,  safety,  and  security 
normally  associated  with  live-fire  training.  RoadRunner  '98  was  sponsored  by  the  Air  Force 
Modeling  and  Simulation  Office  (USAF/XOC)  with  the  Air  Force  Research  Laboratory,  Human 
Effectiveness  Directorate’s  Warfighter  Training  Research  Division  (AFRL/HEA)  serving  as 
program  managers  and  the  Theater  Command  and  Control  Simulation  Facility  (TACCSF)  as 
systems  integrators.  Extensive  support  was  received  from  the  Training  Office  at  Air  Combat 
Command’s  Deputy  for  Operations  (ACC/DOT),  the  Air  Force  Information  Warfare  Center’s 
Advanced  Combat  Simulations  Division  (AFIWC/SAM),  the  Airborne  Warning  and  Control 
Systems  (AW ACS)  training  office  at  the  522nd  Air  Control  Wing,  the  107th  Air  Control  Squadron 
(ACS)  of  the  Iowa  Air  National  Guard  (IANG),  and  the  Naval  Air  Warfare  Center’s  Training 
Systems  Division  (NAWC-TSD).  The  warfighters  who  participated  in  RoadRunner  '98  were 
operational  pilots,  Weapons  Directors  (WDs),  and  Air  Surveillance  Technicians  (ASTs)  from  the 
522nd  ACW,  Tinker  AFB,  OK;  the  27th  Fighter  Wing,  Cannon  AFB,  NM;  the  185th  and  132nd 
Fighter  Wings  of  the  IANG;  the  33rd  Fighter  Wing,  Eglin  AFB,  FL;  and  the  United  States  Air 
Forces  in  Europe  (USAFE). 

This  paper  describes  the  process  by  which  C3  teamwork  dimensions  were  identified,  measures 
formulated,  and  the  rating  instrument  refined.  Measures  of  AWACS  team  performance  were 
developed  based  on  previously  collected  cognitive  task  analysis  data  and  refined  through  the  use 
of  focal  groups  comprised  of  operational  AWACS  WDs  or  WD  Instructors. 
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1.  Distributed  Mission  Training:  Background 

Distributed  simulations  have  become  an  increasingly  essential  aspect  of  operational  military 
training.  As  both  training  and  defense  budgets  shrink,  ways  of  reducing  the  number  of  flight 
hours  and  yet  maintaining  optimal  levels  of  expeditionary  force  readiness  are  sought. 
Distributed  Mission  Training  (DMT)  is  looked  upon  as  a  way  that  will  allow  warfighters  to  train 
in  a  manner  consistent  with  Air  Force  Doctrine  60:  Train  like  you  fight,  fight  like  you  train. 
Previous  constraints  (e.g.,  safety,  flying  status,  flying  hours,  aircraft  and  environmental  factors) 
have  hampered  training  efforts  from  following  this  philosophy  to  its  fullest  capacity.  DMT 
addresses  these  constraints  and  allows  the  warfighters  of  our  nation  to  perform  crucial  training 
missions  with  maximum  proficiency  and  safety,  at  lower  cost.  General  Fogleman,  Chief  of  Staff 
Air  Force,  stated  on  29  Jan  1996: 

/  am  convinced  modeling  and  simulation  technologies  available  today  will 
enable  us  to  significantly  change  the  way  we  train  in  the  future.  We  are  at  a 
crossroads  where  simulator  technology  today  will  be  critical  in  the  success  of 
our  effective  use  of  follow-on  weapon  systems  ....  We  need  to  take  a  hard  look 
at  how  this  technology  will  change  our  training  philosophy  as  well  as  how  we 
develop  future  weapon  systems. 

DMT  is  focused  within  a  synthetic  warfighting  environment  that  grants  a  multitude  of 
advantages  for  more  accurate  observation  and  assessment  in  areas  that  were  previously  difficult 
to  research.  Performance  can  be  modeled  within  and  across  individuals,  teams,  squadrons,  and 
complex  functions.  Linkages  among  diverse  interdependent  units  can  be  identified  and  modeled 
to  achieve  greater  efficiency.  This  intensive  modeling  of  performance  within  a  complex  system 
enables  a  more  precise  assessment  of  the  impact  of  interventions,  such  as  training  technology, 
cause  and  effect  relationships,  and  changes  in  the  distribution  and  display  of  information. 

DMT  offers  flexibility  that  stretches  worldwide  by  linking  participants  across  network  lines, 
allowing  individual  warfighter  training  to  occur  concurrently  in  simulators  whose  presentation  of 
a  commonly  flown  mission  occurs  in  real-time.  This  capability  supplements  cockpit  flight  hours 
with  a  synthetic  reproduction  of  a  complete  and  complex  warfighting  environment  that  combines 
realistic  scenarios  with  simulated  operational  systems.  The  focal  point  of  this  paper  is  the 
Airborne  Warning  and  Control  Systems  (AW ACS)  segment  of  a  Distributed  Mission  Training 
demonstration  (RoadRunner  ’98).  This  exercise  tied  AW  ACS  weapons  directors  from  Tinker 
AFB,  OK,  F-16  aviators  from  AFRL,  AZ,  F-15  aviators  out  of  Kirkland  AFB,  NM,  and  an  A-10 
aviator  from  AFRL,  AZ  together  for  one  week  of  DMT  training. 

2.  USAF  AWACS  Teams 

All  command  and  control  teams  share  some  common  task  and  performance  characteristics.  These 
teams  perform  in  highly  interdependent  roles,  tracking  and  coordinating  some  type  of  tactical 
action,  in  a  manner  consistent  with  overall  strategic  goals  and  procedures,  for  a  defined  sector  of 
air  and/or  land  space  over  a  sustained  period  of  time. 
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The  AW  ACS  team  serves  within  this  team  definition  to  augment  the  need  for  airborne 
surveillance  and  command,  control,  and  communications  functions  for  tactical  and  air  defense 
forces.  This  provides  the  means  to  detect,  identify,  track,  and  intercept  airborne  threats.  The 
aircraft’s  god’s-eye  capability  offers  an  altitude-independent  360°  view  of  more  than  200  miles 
over  both  land  and  water.  There  are  five  mission  crew  categories  related  to  the  function  of  the 
AW  ACS  crew:  the  Mission  Crew  Commander  (MCC),  Senior  Director  (SD),  Weapons  Director 
(WD),  Air  Surveillance  Officer  (ASO),  and  Air  Surveillance  Technician  (AST).  AWACS  teams 
typically  work  in  teams  of  two  to  four  crewmembers  as  part  of  the  airborne  command  and 
control  group  located  within  E-3  Sentry  aircraft. 

The  RoadRunner  '98  exercise  focused  primarily  on  the  integration  of  multiple  warfighter-in-the- 
loop  training  simulators,  combining  them  with  synthetic  forces  to  form  a  virtual  battlespace  to 
conduct  several  assorted  combat-oriented,  tactical  training  exercises.  The  overall  goal  of  this 
effort  was  to  compare  initial  team  performance  with  subsequent  performance  after  participation, 
and  give  the  participants  the  opportunity  to  critique  both  the  systems  and  the  concept  of  training 
(Parton,  1998).  Each  scenario  was  based  on  intelligence  briefings  and  air  tasking  order  (ATO), 
and  each  fighter  and  AWACS  team  planned,  coordinated,  and  briefed  their  mission.  The 
combined  teams  flew  the  mission  in  their  respective  simulators  with  a  trusted  agent  monitoring 
and  evaluating  their  performances.  The  teams  and  instructors  participated  in  an  after-action 
review  aided  by  a  mission  playback  system  that  assisted  during  the  debriefing  process.  Each 
team  participated  in  seven  separate  missions  over  the  course  of  one  week. 

3.  Development  of  Team  Observer  Form 

The  AWACS  crew  coordinates  communications  received  from  a  number  of  sources,  such  as 
other  WDs,  the  SD,  air  surveillance  technicians  (ASTs),  electronic  combat  officers  (ECOs), 
intelligence  operations,  base  operations,  and  friendly  pilots.  To  accomplish  this,  they  must 
exchange,  interpret  and  effectively  weight  information  and  optimize  resource  allocation 
decisions  across  team  members,  over  time,  and  under  stress  and  fatigue.  These  decisions  regard 
shared  resources,  such  as  surface-to-air  missile  sites  and  various  combat,  recomiaissance, 
refueling,  and  search  and  rescue  aircraft.  Relevant  information  must  be  distributed  to 
appropriate  personnel  and  updated  over  time,  in  dynamic  conditions  which  may  require  shift 
changes  in  personnel.  Information  is  often  verbal,  and  may  be  missing,  degraded,  passed  along 
from  unfamiliar  sources,  or  misinterpreted  by  others.  In  addition,  information  is  often 
communicated/interpreted  by  individuals  with  only  partially  overlapping  awareness  of  the 
battlespace.  For  the  RoadRunner  '98  study,  assessment  of  AWACS  team  of  performance  was 
based  on  SME  ratings  of  several  dimensions  of  AWACS  team  performance.  These  dimensions 
were  generated  based  on  cognitive  task  analyses,  followed  with  focus  groups  comprised  of 
experienced  AWACS  instructors.  These  focus  groups  served  to  refine  the  rating  form,  and 
included  the  SMEs  who  actually  participated  as  RoadRunner  '98  trusted  agents/evaluators. 

Cognitive  Task  Analyses 

Assessment  of  individual  and  team  performance  in  realistic  combat  environments  requires  (a)  the 
capability  to  produce  complex  and  dynamic  scenarios,  (b)  identification  of  constructs  that 
represent  important  individual  and  team  skills,  and  (c)  identification  and/or  development  of 
construct  measures.  In  addition,  each  scenario  must  have  identified  quantitative  criteria  of 


87 


mission  success.  Cognitive  task  analyses  informed  and  influenced  the  development  of  scenario 
characteristics  and  the  identification  of  primary  performance  constructs  and  measures. 

Several  cognitive  task  analyses  have  been  performed  on  AW  ACS  operational  personnel.  First, 
analyses  have  been  performed  with  regard  to  AW  ACS  individual  tasks,  with  a  primary  focus  on 
display  enhancements  (Klinger  et  al.,  1993).  In  addition,  Fahey  and  associates  (Fahey  et  al.,  in 
review)  investigated  the  AW  ACS  task  domain  with  a  special  focus  on  AW ACS  team  tasks, 
utilizing  cognitive  task  analysis  and  critical  incident  techniques  for  data  elicitation.  Their  report 
provides  a  comprehensive  description  of  the  AW  ACS  task,  and  the  general  finding  that  critical 
incident  technique  was  quite  useful  for  the  description  of  AW  ACS  teamwork. 

In  addition,  a  follow-up  analysis  of  AW  ACS  tasks,  with  a  primary  focus  on  team  functions  has 
been  articulated  and  initiated  (MacMillan  et  al.,  1998).  This  was  a  preliminary  investigation  to 
test  a  team-based  approach  to  cognitive  task  analysis.  Results  were  very  informative,  supporting 
frameworks  and  constructs  within  an  information-requirements  approach  to  task  descriptions. 

4.  Articulation  of  AW  ACS  Teamwork 

One  core  dimension  that  characterizes  teams  in  general  is  the  type  and  degree  of  interdependence 
among  team  members  (Saavedra,  et  al.,  1993;  Salas,  et  al.,  1992).  Teams  are  distinguished  from 
groups  in  general  by  a  common  purpose  or  goal,  performed  by  interdependent  team  members 
(Salas,  et  al.,  1992).  From  this  definition  we  derived  a  core  definition  of  teamwork:  The 
fundamental  function  of  teamwork  is  the  effective  managing  of  interdependencies  to  accomplish 
a  team  goal.  From  this  core  definition  we  identified  six  dimensions  of  team  interdependence 
(Elliott  &  Schiflett,  in  review). 

Discussion  with  focal  groups  comprised  of  AWACS  subject-matter  experts  led  to  refinement  of 
our  preliminary  team  task  taxonomy,  as  applied  to  AWACS  team  training  and  performance 
evaluation.  Prior  to  the  RoadRunner  '98  exercise,  we  interviewed  38  expert  AWACS  weapons 
directors.  These  groups  reviewed  the  proposed  scenarios  and  assisted  in  the  evaluation  and 
refinement  of  an  instructor/observer  rating  form  (see  Appendix).  Review  of  the  initial 
taxonomy,  which  was  generated  to  classify  teams  in  general,  led  to  the  identification  of  four 
teamwork  functions  which  are  specific  to  AWACS  teamwork:  (a)  mission  planning,  (b) 
communication  content/timing,  (c)  adherence  to  communication  protocol,  and  (d)  supporting 
behavior. 

4.1  Mission  planning:  formulation  of  “contracts” 

AWACS  perfonnance  can  be  divided  into  three  phases:  premission  planning,  task  execution,  and 
debriefing  (discussion  after  task  execution).  Mission  planning  affects  AWACS  teamwork  by 
establishing  roles,  responsibilities,  and  contingency  plans.  As  discussed  by  Fahey  et  al.  (1998) 
and  Macmillan  et  al.  (1998)  AWACS  weapons  directors  explicate  roles,  responsibilities,  and 
strategies  to  manage  team  member  interdependencies  through  the  establishment  of  “contracts.” 
These  contracts  are  made  among  AWACS  team  members  (internal  contracts),  and  to  the 
“external”  team  (i.e.,  pilots  of  friendly  assets). 
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4.2  Communication:  adherence  to  protocol 

AW  ACS  weapons  director  tasks  are  based  predominately  on  the  exchange  of  verbal  information. 
Communications  are  heavily  standardized  in  terms  of  content  (jargon)  and  process.  This  aspect 
of  communication  effectiveness  refers  to  the  degree  to  which  individuals  follow  guidelines  for 
communication  exchange.  In  addition  to  proper  jargon  and  syntax,  communications  must  be 
clear,  concise,  and  correct. 

4.3  Communication  to  support  situation  awareness 

Communications  may  be  clear,  concise,  correct,  and  delivered  according  to  proper  protocol,  and 
still  be  superfluous.  The  maintenance  of  situation  awareness  also  requires  that  pertinent 
information  be  exchanged  to  the  right  person,  at  the  right  time.  In  such  a  communication-rich 
enviromnent,  too  much  communication  can  impede  performance,  when  unnecessary  information 
“steps  over”  other,  more  urgent  communication.  Indeed,  part  of  the  proper  timing  of  AWACS 
communications  to  pilots  involves  knowledge  of  when  to  keep  quiet. 

4.4  Supporting  behavior 

While  it  would  seem  apparent  that  this  aspect  is  most  clearly  representative  of  teamwork,  the 
conceptualization  of  supporting  behavior  among  AWACS  weapons  directors  was  more  difficult 
to  validate  through  subject-matter  experts.  While  there  was  no  doubt  that  AWACS 
teammembers  do  support  each  other,  the  argument  was  made  that  most  of  the  support  is  in  the 
form  of  communication  exchange.  AWACS  teammembers  support  each  other  primarily  through 
updates  and  reminders  of  salient  information.  At  the  same  time,  they  can  also  transfer  resources 
(responsibilities)  and  confer  on  decisions/actions,  therefore,  it  was  decided  to  keep  this  construct 
as  an  independent  aspect  of  AWACS  teamwork. 

5.  Rating  Instrument 

Once  the  dimensions  of  AWACS  teamwork  performance  were  refined  by  the  SME  focus  groups, 
the  rating  instrument  was  tailored  to  capture  these  dimensions  across  three  phases  of 
performance:  (a)  premission  briefing,  (b)  mission  performance,  and  (c)  mission  debriefing. 
Ratings  were  based  on  a  4-point  scale,  using  traditional  rating  assessment  categories.  The  rating 
categories  used  are  based  on  the  same  categories  of  performance  used  in  AWACS  WD  training: 


AWACS  Team  Observer  Form 
Rating  and  Definition  of  Performance  Criteria 

1.  No  Ability  or  Knowledge.  Task  Failure 

2.  Lacks  Proficiency.  (Coordination,  Communication,  Cohesion) 

3.  Uncorrected  errors.  Degraded  Mission  Outcome  or  Endangered  Friendly  Forces 

4.  Limited  Proficiency.  Recognizes  and  corrects  errors  with  team  recovery.  Mission  Degraded 

5.  Proficient.  No  mission  impacting  errors.  Team  reacts  correctly  in  current  situations 

6.  Highly  Proficient.  Prevents  errors.  Team  anticipates  future  situations.  Plans  ahead. 

NA  =  Not  Applicable  NO  -  Not  Observed  NP  =  Not  Performed 

Table  1:  Rating  Scale  for  Dimensions  of  Performance 
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WDs  were  rated  on  the  following  during  the  Phase  I  portion  of  their  performance  evaluation: 

(a)  Development  of  mission  aids, 

(b)  Formulation  of  contracts  internal  to  the  AW  ACS  team, 

(c)  Formulation  of  contracts  external  to  the  AW  ACS  team,  and 

(d)  Prebrief  of  pilots. 

These  aspects  were  identified  during  the  subject-matter  expert  focal  interviews  as  critical  to 
successful  premission  planning  and  team  performance.  The  development  of  mission  aids 
included  the  generation,  refinement,  and  review  of  the  communications  worksheet  (which 
specifies  who  talks  to  whom  on  which  channel),  the  fighter  flow  sheet,  the  chart,  and  the  fact 
sheet,  all  of  which  describes  and/or  specifies  procedures  to  enhance  awareness  and  coordination. 
Formulation  of  internal  contracts  refers  to  the  agreements  made  among  the  WDs  as  to  their  roles 
and  responsibilities.  Communication  tasks  and  console  assignments  are  set,  specific  mission 
objectives  are  discussed,  and  functional  responsibilities  and  contingency  plans  are  specified  and 
assigned.  Formulation  of  external  contracts  refers  to  the  procedures  executed  between  the  WDs 
and  others  (pilots)  with  regard  to  roles  and  responsibilities,  such  as  those  regarding  objectives, 
strategies,  and  communications.  These  contracts  are  then  discussed  within  the  briefing  session 
{Pre-brief  Pilots)  by  the  lead  WD  with  the  lead  pilot,  along  with  issues  such  as  the  rules  of 
engagement  (ROE)  and  Air  Tasking  Order  (ATO)  compliance. 

Phase  II  performance  (during  mission  execution)  was  assessed  by  ratings  regarding  (a) 
communication  in  accordance  with  3-1  and  Unit  Standards,  (b)  communications  in  support  of 
situation  awareness  and  the  “big  picture”,  and  (c)  mission  execution.  Communications  are 
assessed  in  terms  of  adherence  to  established  protocol,  which  strives  to  maximize  aspects  of 
clarity,  brevity,  and  accuracy — communications  that  are  clear,  concise,  and  correct.  This 
includes  communications  among  WDs  and  also  their  communications  to  others  (rated 
separately).  In  addition,  communications  are  also  rated  with  regard  to  relevance  and 
timeliness — is  the  information  what  was  needed  to  maintain  team  situation  awareness — was  the 
right  information  “pushed”  to  the  right  person  at  the  right  time?  Mission  execution  was  assessed 
through  consideration  of  ATO  execution,  contract  execution  (formulated  during  premission 
briefing),  and  adaptive  replanning  (flexibility,  problem-solving,  contingency  generation  as 
needed). 

Phase  III  (mission  debriefing)  performance  was  assessed  through  ratings  of  (a)  reconstruction  of 
the  engagement,  (b)  evaluation  of  team  objectives,  (c)  review  of  equipment  issues,  (d)  review  of 
team  mission  execution,  and  (e)  review  of  information  exchange.  Reconstruction  of  the 
engagement  was  assessed  through  consideration  of  the  process  of  reviewing  the  recordings  and 
the  identification  of  conflicts  and/or  problems.  Objectives  were  reviewed  in  light  of  mission 
support  and  execution  of  contracts.  Equipment  issues  included  communication,  console 
assignments,  and  any  failures/alibis.  Team  mission  execution  was  assessed  through 
consideration  of  priority  objectives,  training  objectives,  and  any  failures/alibis.  Information 
exchange  was  considered  with  regard  to  internal  and  also  external  communications. 
Communications  should  have  supported  overall  situation  awareness,  and  lessons  learned  should 
be  identified  during  the  mission  debriefing. 
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Each  team  was  rated  on  various  dimensions  of  performance  across  three  phases  of  performance, 
for  each  mission.  They  were  assessed  for  performance  in  Phase  I  (premission  briefing),  Phase  II 
(mission  execution),  and  Phase  III  (mission  debriefing).  Measures  and  variable  names  are  as 
follows: 


5.1  Premission  Briefing 

Development/Use  of  Mission  Aids  (MSNAID  l):  Effective  use  of  appropriate  mission 
planning  aids,  such  as  communications  worksheets,  charts,  fighter  flow  sheet,  and  fact  sheet. 

Negotiation  of  Contracts  Internal  to  AW  ACS  Team  (ICONTR_l):  Effective  discussion 
and  coordination  of  interdependencies  among  AWACS  WDs  and  AST,  with  regard  to 
communications,  console  setup,  mission  tasks  (who  is  primary/assist;  coordination  with  AST), 
and  mission  objectives. 

Negotiation  of  Contracts  External  to  AWACS  Team  (ECONTR_l):  Effective  discussion 
and  coordination  with  pilots,  with  regard  to  communications,  mission  tasks,  and  objectives. 

Pre-brief  with  Pilots  (PREVI  1).  Effective  discussion  and  coordination  with  pilots,  with 
regard  to  objectives,  standards  (Wing  and  Squadron),  Air  Tasking  Order  compliance,  and  Rules 
of  Engagement. 

Overall  for  Premission  Briefing  (PHASEI).  Average  of  above.  Measures. 


5.2  Mission  Execution 

Communications  in  accordance  with  Standards.  Internal  to  AWACS  Teams  (COMMI_2): 
Extent  of  compliance  to  standard  communication  protocols,  regarding  continuum  of  control, 
Land  Unit  Standards,  C-3  (Clear,  Concise,  Correct),  and  use  of  correct  call  signs. 

Communications  in  accordance  with  Standards.  External  to  AWACS  Teams 
(COMME_2):  Extent  of  compliance  to  standard  communication  protocols,  regarding  continuum 
of  control,  Land  Unit  Standards,  C-3  (Clear,  Concise,  Correct),  and  use  of  correct  call  signs. 

Maintenance  of  Situational  Awareness  and  Big  Picture  in  Communications  Internal  to 
AWACS  Team  ISA  PI  21:  Content  and  timing  of  communications  (as  opposed  to  process). 

Maintenance  of  Situational  Awareness  and  Big  Picture  in  Communications  External  to 
AWACS  Team  (S A  PE  21:  Content  and  timing  of  communications  (as  opposed  to  process). 

Mission  Execution  internal  to  AWACS  Team  (MSNEXI  2):  Performance  with  regard  to 
execution  of  Air  Tasking  Order,  Contract  execution,  and  adaptive  replanning  (flexibility, 
problem  solving,  and  contingency). 

Mission  Execution  External  to  AWACS  Team  (MSNEXE_2):  Performance  with  regard 
to  execution  of  Air  Tasking  Order,  Contract  execution,  and  adaptive  replanning  (flexibility, 
problem  solving,  and  contingency). 

Overall  mission  execution  (PHASEII):  average  of  six  measures  above. 


5.3  Team  Debrief 

Reconstruction  of  Engagement-Internal  to  AWACS  Team  (RECENG  3):  Effectiveness  of 
review  of  engagement  through  review  of  recording  and  identification/discussion  of 
conflicts/problems . 

Evaluation  of  Performance  with  regard  to  Objectives  Internal  to  AWACS  Team 
(EVALON  3):  Effectiveness  of  review  of  performance  with  regard  to  mission  support  and 
contracts. 
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Equipment-Internal  to  AW  ACS  Team  (EQUEP3):  Effective  identification  and  discussion 
of  issues  with  regard  to  equipment,  such  as  communication  and  checkout,  console  assignments  and 
check-outs,  and  failures/alibis. 

Discussion  of  Team  Mission  Execution  External  to  AWACS  Team  (TMISEX  3). 
Identification  and  discussion  of  issues  with  regard  to  support  of  priority  objectives,  WD/AST 
training  objectives,  and  failures/alibis. 

Information  Exchange  Internal  to  AWACS  Team  (INFEXI3).  Effective  discussion  of 
information  exchange  among  AWACS  team  members,  with  regard  to  maintenance  of  shared 
mental  picture  (situation  awareness)  and  lessons  learned. 

Information  Exchange  External  to  AWACS  Team  (1NFEXE  3).  Effective  discussion  of 
information  exchange  among  AWACS  team  members  and  others,  with  regard  to  maintenance  of 
shared  mental  picture  (situation  awareness)  and  lessons  learned. 

Overall  Mission  Debriefing  (PHASEIII).  Average  of  six  measures  above. 

Overall  Mission  Performance  (OVERALL).  Average  of  all  measures. 


6.  Results 


Descriptive  Statistics:  Average  ratings  across  teams  (3),  scenarios  (7),  and  raters  (2  per  team  and 
scenario  for  Phase  II).  Results  indicate  variance  in  distribution  in  ratings,  with  utilization  of  the 
entire  rating  scale,  for  most  measures. 


Pretnission  Planning:  N 
MSNAID_1  25 

ICONTRl  25 

ECONTRl  24 

PREVI  1  24 


Minimum  Maximum 


Debrief: 

RECENG3 

EVALON3 

EQUIP3 

TMISEX_3 

INFEXI3 

INFEXE_3 

PHASE  III 


24 

24 

26 

24 

25 
24 
27 


2.00 

2.00 

1.50 

1.00 


4.00 

4.00 

4.00 

4.00 


Mean 

3.0800 

2.9600 

2.8125 

2.8542 

2.9200 


2.00 

2.00 

2.00 

1.00 

1.00 

1.00 

1.50 


4.00 

4.00 

4.00 

4.00 

4.00 

4.00 

4.00 


3.1250 

3.0417 

3.1154 

3.0833 

3.0400 

2.9792 

3.0562 


Std.  Deviation 
.4933 
.6758 
.6726 
.6833 
.4703 


Task  Execution: 

COMMI  2 

42 

1.00 

4.00 

2.9881 

.7690 

COMME  2 

42 

1.00 

4.00 

2.7143 

.7741 

SA  PI  2 

42 

1.00 

4.00 

2.9524 

.7949 

SA  PE  2 

42 

1.00 

4.00 

2.6905 

.8762 

MSNEXI  2 

42 

1.00 

4.00 

3.0000 

.7730 

MSNEXE  2 

41 

1.00 

4.00 

2.6951 

.9413  j 

PHASE  II 

42 

1.33 

4.00 

2.8421 

.7301 

.6124 

.7929 

.4961 

.8805 

.8406 

.8140 

.5970 


OVERALL _ 22 _ 1.94  3.69 _ 2.8699 _ .4705 

Valid  N  (listwise)  16 
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6.1  Differences  in  performance  attributable  to  differences  among  raters. 

ANOVA  analyses  indicated  no  significant  differences  among  raters  with  regard  to  any  of  the 
measures.  In  addition,  regression  analyses  based  on  the  consideration  of  trusted  agents,  teams, 
and  scenarios  indicate  that  the  variance  attributable  to  differences  among  trusted  agents  was  not 
significant. 

6.2  Relative  impact  of  three  sources  of  variance  (trusted  agents,  teams,  and  scenarios). 

There  were  three  primary  sources  of  variance:  that  attributable  to  raters  (which  should  be 
minimized),  to  differences  among  teams  (not  of  primary  interest  here),  and  to  the  training 
exercise  itself  (differences  among  the  seven  scenarios  was  predicted  to  demonstrate 
improvement  over  time).  Multiple  regression  analyses  were  performed,  first  on  the  model 
composed  of  trusted  agents,  teams,  and  scenarios.  This  model  predicted  73%  of  the  total 
variance  in  ratings.  In  addition,  the  consideration  of  trusted  agents  did  not  add  to  the  prediction 
of  ratings,  thus  indicating  reliability  of  measures  with  regard  to  raters.  The  analyses  were  then 
run  using  teams  and  scenarios  as  predictors,  and  the  model  still  predicted  73%  of  the  total 
variance  in  ratings. 

Model  Summary:  Predictors:  (Constant),  SCENARIO,  TEAM,  TRUSTEDA 
Dependent  Variable:  OVERALL 


Model  R 

R  Square 

Adjusted  R  Square  Std.  Error  of  the  Estimate 

1  .859 

.738 

.694 

.2603 

ANOVA 

Sum  of  Squares  df 

Mean  Square 

F 

Sig. 

1  Regression 

3.429 

3 

1.143 

16.867 

.000 

Residual 

1.220 

18 

6.776E-02 

Total 

4.648 

21 

Coefficients 

Coef-B 

S.Err 

Beta 

t 

Sig. 

1  (Constant) 

2.591 

.221 

11.750 

.000 

TRUSTEDA 

-4.394E-02 

.045 

-.117 

-.969 

.345 

TEAM 

-.170 

.070 

-.294 

-2.436 

.025 

SCENARIO 

.177 

.027 

.792 

6.557 

.000 

a  Dependent  Variable:  OVERALL 

These  data  indicate  that  participation  in  the  training  exercise  had  a  significant  effect  on 
performance,  across  all  teams,  and  for  most  measures  of  performance  across  the  three  phases  of 
performance.  Results  are  described  in  further  detail  for  the  effects  due  to  differences  among  teams, 
and  differences  across  scenarios. 

6.3  Differences  in  performance  attributable  to  differences  among  teams. 

Data  demonstrate  the  advantage  in  careful  construction  of  performance  constructs.  The  overall 
measure  of  performance  was  based  on  the  mean  of  all  measures;  ANOVA  results  did  not  indicate 
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significant  differences  among  teams.  There  were,  however  differences  among  teams  for  the 
measurement  of  one  aspect  (negotiation  of  contracts  among  team  members)  of  performance  during 
premission  briefing,  and  for  nearly  all  aspects  of  performance  during  mission  execution.  Teams 
did  not  differ  significantly  in  performance  during  mission  debriefing.  An  overall  assessment  of 
performance  may  be  too  generic  to  capture  effects  of  a  predictor  variable. 

ANOVA:  Significant  Differences  By  Team,  Between  Groups 


Variable 

Sum  of  Squares 

df 

MnS< 

ICONTR_l 

3.363 

2 

1.681 

MSNEXE 2 

5.730 

2 

2.865 

SA  PE  2 

4.298 

2 

2.149 

SA_PI_2 

4.762 

2 

2.381 

COMME_2 

3.857 

2 

1.929 

COMMI2 

2.869 

2 

1.435 

PHASE2 

3.298 

2 

1.649 

869 

665 


3.083 

4.392 

3.631 

2.617 

3.466 


.057* 

.019** 

.036** 

.086* 

.041** 


6.4  Differences  attributable  to  scenarios  and  experience. 

Team  performance  differed  significantly  across  scenarios  for  many  of  the  measures  (See 
Appendix  for  descriptives  and  ANOVA  output).  Differences  in  measures  among  scenarios 
would  capture  differences  due  to  the  scenario  content,  as  scenarios  differed  in  mission  objective 
and  content.  They  would  also  be  due  to  experience  over  time.  However,  Scenario  1  and 
Scenario  7  are  very  similar  in  content,  to  better  ascertain  the  effect  of  experience  on 
performance.  First  we  describe  differences  across  all  scenarios.  The  following  graph  describes 
overall  results  of  instructor  ratings  of  AWACS  team  performance,  for  each  mission  session, 
ordered  over  time. 


The  above  graph  provides  a  succinct  glimpse  of  trends  regarding  the  effect  of  simulation 
participation  on  performance,  through  differences  in  performance  from  Scenario  1  and  Scenario 
7,  by  phase.  Here  it  can  be  seen  that  variance  in  performance  across  scenarios  was  more 
attributable  to  performance  in  Phases  II  and  III:  teams  were  lower  in  performance  during  Phase 
II  and  III  during  the  first  scenario,  and  demonstrated  higher  improvement  in  these  phases.  At  the 
same  time,  performance  during  Phase  I  (premission  briefing)  also  improved  over  time. 

Overall  results  for  each  team  for  Scenario  1  (pretest)  and  Scenario  7  (post-test).  Team  A  was  the 
highest  performing  team  for  both  pre-and  post  scores.  Improvement  for  this  team  was  consistent 
across  the  three  phases.  In  comparison,  Teams  B  and  C  performed  less  well  from  the  start,  and 
particularly  less  well  in  Phase  II  (mission  performance)  and  Phase  III  (debriefing  perfonnance). 
Data  indicate  that  training  needs  exist  for  the  kind  of  training  that  distributed  mission  training 
provides:  improvement  in  team  premission  briefing,  team  coordination  and  team  debriefing.  In 
addition,  data  demonstrate  the  effectiveness  of  participation  in  the  training  exercise  for  the 
improvement  of  performance  across  all  phases  of  each  mission. 
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7.  Discussion 


Results  were  consistent  with  expectations.  Simulation-driven  distributed  mission  training  was 
expected  to  add  value  through  the  experience  of  complexity  of  team-on-team  interdependencies. 
The  immersion  of  individuals  into  teams  within  a  global  mission  scenario  should  enhance  their 
capability  of  managing  these  interdependencies  during  performance.  Debriefing  experience  is 
particularly  valuable  to  AW  ACS  crew  as  they  often  do  not  get  the  advantage  of  debriefing  with 
pilots  after  the  mission;  their  mission  usually  extends  beyond  the  time  of  perfonnance  of  a 
particular  combat  aircraft.  The  value  of  this  experience  is  indicated  by  the  increased 
performance  during  debriefing  of  later  mission  scenarios.  In  addition,  teams  improved  on  most 
aspects  of  performance  throughout  mission  planning  and  execution. 

Data  demonstrate  the  advantage  of  careful  construction  of  performance  constructs.  The  overall 
measure  of  performance  was  based  on  the  mean  of  all  measures;  ANOVA  results  did  not  indicate 
significant  differences  among  teams.  There  were,  however  differences  among  teams  for  the 
measurement  of  one  aspect  (negotiation  of  contracts  among  team  members)  of  performance 
during  premission  briefing,  and  for  nearly  all  aspects  of  performance  during  mission  execution. 
Teams  did  not  differ  significantly  in  performance  during  mission  debriefing.  An  overall 
assessment  of  performance  may  be  too  generic  to  capture  effects  of  a  predictor  variable. 

8.  Summary 

The  following  quote  very  aptly  captures  the  essence  of  the  RoadRunner  '98  experience: 

The  RoadRunner  '98  concept  of  operations  and  its  execution  was  a  valuable  tool  in 
development  and  evolution  of  Distributed  Mission  Training  (DMT).  This  exercise 
demonstrated  the  ability  to  interface  multiple  virtual  warfighters  and  synthetic  forces  to  form 
a  virtual  battlespace  that  can  be  used  to  conduct  realistic  combat-oriented  training.  Our 
goal  was  to  compare  team  performance  before  and  after  DMT  training  and  to  give  the 
participants  the  opportunity  to  critique  both  the  systems  and  this  concept  of  training.  The 
initial  comments  from  the  AWACS  crews  suggest  that  they  found  DMT  to  be  a  viable  and 
productive  training  tool.  The  technology  was  proven  capable  of  providing  combat-oriented 
training  at  low  cost,  with  fewer  risks,  while  still  maintaining  the  security  constraints 
associated  with  aviation  duties.  RoadRunner  '98  has  shown  the  potential  to  provide 
improvements  in  pre-planning  and  mission  execution  with  emphasis  in  shared  training 
exercises  to  provide  endless  training  opportunities  to  the  21s‘  century  aviator. 

Randy  L.  Parton,  Major,  USAF 
Chief  Modeling  and  Simulation 

The  approach  taken  during  the  evolution  of  the  rating  scales  enabled  the  collection  of  data  that 
significantly  adds  to  the  understanding  and  research  of  team  communications,  interdependencies, 
and  training. 

Post-mission  questionnaires  provided  comments,  information,  and  guidance  that  have  proves 
helpful  in  the  creation  of  continued  DMT  missions  and  research. 
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Participants  responded  to: 

How  mission  impacted  learning  and  training: 

•  A  staged  4-day  war  would  be  very  beneficial. 

•  Using  voice/data  recording  capability  helps  us  pinpoint  errors  and  allows  a  thorough 
debrief,  enhancing  the  learning  experience. 

Whether  exercise  helped  meet  squadron  training  objectives: 

•  Good  intro  to  a  Red  Flag  exercise. 

•  Great  real  war  training,  with  inexperienced ...  benefiting  most. 

Whether  exercise  has  the  potential  of  meeting  squadron  training  objectives: 

•  Work  the  bugs  and  this  could  be  a  simulated  Red  Flag. 

•  It  will  definitely  increase  amount  of  learning  gained from  SIM. 

Suggestions  for  improvements: 

•  Tie  SIMs  together  with  a  grand  order  of  battle,  Intel,  and  more  complete  mission 
planning. 

•  Allow  full-crew  participation. 

•  Anything  that  can  go  wrong  on  the  jet  needs  to  be  able  to  go  wrong  here. 

In  addition,  participants  commented  on  the  number  of  “lessons  learned”  that  occur  when  training 
involves  so  much  training  in  one  week,  and  believed  the  non-predictability  of  real  pilots  manning 
the  aircraft,  as  opposed  to  sim  drivers,  added  indescribably  to  the  realism  of  the  mission. 

The  ongoing  development  of  distributed  mission  training  will  allow  USAF  to  move  into  the  next 
century  as  a  cohesive,  well  trained  warfighting  unit.  The  technology  and  research  providing  the 
foundation  for  this  capability  will  allow  for  a  more  efficient,  cost-effective,  and  objective  means 
of  training  that  has  only  begun  to  make  its  impact  on  the  military  concept  of  warfighter 
readiness. 
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